mirror of
https://github.com/GSA/notifications-api.git
synced 2026-08-18 13:38:53 -04:00
Updates to the delete CSV file job to reduce the number of eligible jobs in any run
- previously this was unbounded, so it got all jobs older then 7 days. In excess of 75,000 🔥
- this meant that the job took (a) a long time and (b) a lot memory and (c) doing the same thing every day
These changes mean that the job has a 2 day eligible window for jobs, minimising the number of eligible jobs in a run, whilst still retaining some leeway in event if it failing one night.
In principle the job runs early morning on a given day. The previous 7 days are left along, and then the previous 2 days worth of files are deleted:
so:
runs on
31st
30,29,28,27,26,25,24 are ignored
23,22 jobs here have files deleted
21 and earlier are ignored.
This commit is contained in:
@@ -116,7 +116,8 @@ def dao_update_job(job):
|
||||
db.session.commit()
|
||||
|
||||
|
||||
def dao_get_jobs_older_than(limit_days):
|
||||
def dao_get_jobs_older_than_limited_by(older_than=7, limit_days=2):
|
||||
return Job.query.filter(
|
||||
cast(Job.created_at, sql_date) < days_ago(limit_days)
|
||||
cast(Job.created_at, sql_date) < days_ago(older_than),
|
||||
cast(Job.created_at, sql_date) >= days_ago(older_than + limit_days)
|
||||
).order_by(desc(Job.created_at)).all()
|
||||
|
||||
Reference in New Issue
Block a user