darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2026-07-18 05:30:48 -04:00
Code Issues Packages Projects Releases Wiki Activity

set processing_started in before earlier jobs are processed

Browse Source 
process_incomplete_jobs loops through jobs processing them in a single
task. This means that if the job statuses are all 'error', and then the
process_incomplete_jobs task fails, the later jobs in the list that
never got picked up won't have their status set back to in progress, or
their processing_started time - so will be stuck in 'error' forever.

Instead, we set the job statuses to in progress and the start time to
now before we process any - so if the incomplete_jobs task fails later,
the jobs will be picked up (again) by the check_job_statuses task in
half an hour's time
This commit is contained in:
Leo Hemsted
2018-03-09 17:16:48 +00:00
parent 64bb94af9e
commit ea2b0dfbc9
2 changed files with 43 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
app/celery/tasks.py
View File

@@ -550,6 +550,14 @@ def send_inbound_sms_to_service(self, inbound_sms_id, service_id):
@notify_celery.task(name='process-incomplete-jobs')
@statsd(namespace="tasks")
def process_incomplete_jobs(job_ids):
jobs = [dao_get_job_by_id(job_id) for job_id in job_ids]
# reset the processing start time so that the check_job_status scheduled task doesn't pick this job up again
for job in jobs:
job.job_status = JOB_STATUS_IN_PROGRESS
job.processing_started = datetime.utcnow()
dao_update_job(job)
current_app.logger.info("Resuming Job(s) {}".format(job_ids))
for job_id in job_ids:
process_incomplete_job(job_id)
@@ -559,11 +567,6 @@ def process_incomplete_job(job_id):
job = dao_get_job_by_id(job_id)
# reset the processing start time so that the check_job_status scheduled task doesn't pick this job up again
job.job_status = JOB_STATUS_PENDING
job.processing_started = datetime.utcnow()
dao_update_job(job)
last_notification_added = dao_get_last_notification_added_for_job_id(job_id)
if last_notification_added: