We now have a new column in the database, but it isn't being
populated. The first step is to make sure we update this column,
while still keeping the old enum based column up to date as well.
A couple of changes have had to happen to support this - one irritating
thing is that if we're ever querying columns individually, including
`Notification.status`, then we'll need to give that column a label,
since under the hood it translates to `Notification._status_enum`.
Accessing status through the ORM (i.e., my_noti.status = 'sending' or
similar) will work fine.
The is_between_end_date_exclusive is a bit funny.
Perhaps the better way to handle it is to make the function is_between but change the financial year function return an enddate that is one millisecond less. That way we can always use the between logic and it will be easier to use.
Since the response has changed I have created new endpoints so that the deployments for Admin are more managable.
Removed print statements from some tests.
Once we have the new columns in notifications table, the query will need to include the rate multiplier and if the number is international.
The monthly billing query will be built next.
We only need one rate per channel. This reflects that. The provider_rates has been left for now, it is still not being used.
New dao has been added to select the right rate for the given notification_type and date of notificaiton.
- previously this was unbounded, so it got all jobs older then 7 days. In excess of 75,000 🔥
- this meant that the job took (a) a long time and (b) a lot memory and (c) doing the same thing every day
These changes mean that the job has a 2 day eligible window for jobs, minimising the number of eligible jobs in a run, whilst still retaining some leeway in event if it failing one night.
In principle the job runs early morning on a given day. The previous 7 days are left along, and then the previous 2 days worth of files are deleted:
so:
runs on
31st
30,29,28,27,26,25,24 are ignored
23,22 jobs here have files deleted
21 and earlier are ignored.
If the date range is with the last 7 days we query Notifcations.
If the date range is outside of the last 7 days we query NotificationHistory.
NotificationHistory does not persist notifications created with a test api key.
This will transform each notification in a job to a row in a file.
The file is then uploaded to S3.
The files will later be aggregated by the notifications-ftp app to send to dvla.
The method to upload the file to S3 should be pulled into notifications-utils package.
It is the same method used in notifications-admin.
