reset job processing time when re-processing incomplete jobs

we might stop processing jobs mid-way through if, for example, a
deploy or downscale kills the box working on it. We have a scheduled
task that identifies any job that we started processing more than half
an hour ago that is still processing.

However, we encountered a bug where we triggered the
process_incomplete_job multiple times, because the processing_started
of the job was still set to half an hour ago. If we reset the
processing_started to the current time, then it won't get picked up by
future runs of the check_job_status scheduled task.
This commit is contained in:
Leo Hemsted
2018-03-09 16:30:50 +00:00
parent c7cc7822f7
commit f0ca3d40de
3 changed files with 53 additions and 0 deletions

View File

@@ -559,6 +559,10 @@ def process_incomplete_job(job_id):
job = dao_get_job_by_id(job_id)
# reset the processing start time so that the check_job_status scheduled task doesn't pick this job up again
job.processing_started = datetime.utcnow()
dao_update_job(job)
last_notification_added = dao_get_last_notification_added_for_job_id(job_id)
if last_notification_added: