- previously this was unbounded, so it got all jobs older then 7 days. In excess of 75,000 🔥
- this meant that the job took (a) a long time and (b) a lot memory and (c) doing the same thing every day
These changes mean that the job has a 2 day eligible window for jobs, minimising the number of eligible jobs in a run, whilst still retaining some leeway in event if it failing one night.
In principle the job runs early morning on a given day. The previous 7 days are left along, and then the previous 2 days worth of files are deleted:
so:
runs on
31st
30,29,28,27,26,25,24 are ignored
23,22 jobs here have files deleted
21 and earlier are ignored.
If the date range is with the last 7 days we query Notifcations.
If the date range is outside of the last 7 days we query NotificationHistory.
NotificationHistory does not persist notifications created with a test api key.
This will transform each notification in a job to a row in a file.
The file is then uploaded to S3.
The files will later be aggregated by the notifications-ftp app to send to dvla.
The method to upload the file to S3 should be pulled into notifications-utils package.
It is the same method used in notifications-admin.
* Add notify user id in config
* Add dao method to get provider history versions along with tests
* BUG: Provider switching did not handle case where priorities were equal. This
* adds a fix to properly cover this case along with tests
By removing the join to services for the fetch_stats_by_date_range_for_all_services and dao_fetch_todays_stats_for_all_services
queries we get a 30% and 25% performance improvement.
until work is done to stop using PUT /user/{id} on the admin app, this
function also needs to reset failed logins, cos it's used during the
forgotten password flow
in verify_user_password, if succesful we reset the failed_login_count.
now we use failed_login_count for 2FA attempts, we need to make sure we
reset it in other places too, so that people don't get blocked,
especially in the reset-password user journey.
* verify_user_code - if it's succesful, reset the failed_login_count
* update_password - reset failed_login_count because either
* you're logged in and so it's 0 anyway
* you're resetting your password via pword reset link, and the old
count isn't relevant anymore
When the verify code is wrong or expired increment the failed to login count for the user.
When the verify code is successfully used reset the failed login count to 0.
Cache expires every 10 minutes, but will help with the every 2 second query, especially when a job is running.
There is some clean up and qa to do for this yet
