Stopping celery via upstart issues a TERM signal to the whole process group, which immediately kills the worker threads. This can lead to inconsistentcy in the state of a notification as an API call, for example, can take several seconds consequently the likelihood of the call being lost due to a restart is relatively high.
We need to ensure that we stop cleanly to ensure code deploys and autoscale events do not cause issues.
this script is called as part of a pre-stop hook in upstart, which passes a pid file location to the script. The script will then issues the TERM signal to only the master celery process. which will cleanly shut down the workers and exit.
the script will loop based on the existence of a /proc/{pid} for the celery master process. Once the file is gone the script will exit. Upstart blocks on this script before issuing kill on whatever processes may be left hanging around.
the length of the loop is a maximum 15 minutes - which is the tested max time a task may take. In reality it will exit much quicker.
- brings boto S3 into new AWS folder
- CSV processing utils method
Rejigs the jobs rest endpoint - removes some now unused endpoints,
Calls to the task with the job, job processing in task, delegating SMS calls to the sms task
- uses the new subject/email from fields present on the templates / service tables
- brings the send email api call into line with the sms one.
- same fields (to/template_id)
- same rules regarding restricted services
- wired in as a task into celery
Requires
- new celery queue
- new env property (NOTIFY_EMAIL_DOMAIN)
