Previous:
assumed discrepancy in stats counts to be related to timeouts
Now:
If discrepancy exists do the math on the notifications for that job to work out counts based on statuses to redo stats.
After three days we timeout the notifications that we have not received a receipt for.
In the same way we bump the failed count to match the job count if there is a descepenecy. We do this after the same period that we do for notifications.
- Looks now for a specific Integrity error, not a generic SQLAlchemy error when checking for an insert failure
- If the update fails post an insert error, then there is an issue so raise an exception so tasks can be retried.
- create_or_update_job_sending_statistics
This will try and update an existing row. if this fails as it hasn't been created, then it will insert a row. If this fails as another process has got there first then it should try and update again.
This is a code version of upset
- update_job_stats_outcome_count
Will update the outcome states. Uses the NOTIFICATION_STATUS_TYPES_FAILED to determine if the notification failed.
Any thing in DELIVERED will be marked as delivered. Statues not in the FAILED array or delivered provoke no update to the table.
We now have a new column in the database, but it isn't being
populated. The first step is to make sure we update this column,
while still keeping the old enum based column up to date as well.
A couple of changes have had to happen to support this - one irritating
thing is that if we're ever querying columns individually, including
`Notification.status`, then we'll need to give that column a label,
since under the hood it translates to `Notification._status_enum`.
Accessing status through the ORM (i.e., my_noti.status = 'sending' or
similar) will work fine.
The is_between_end_date_exclusive is a bit funny.
Perhaps the better way to handle it is to make the function is_between but change the financial year function return an enddate that is one millisecond less. That way we can always use the between logic and it will be easier to use.
Since the response has changed I have created new endpoints so that the deployments for Admin are more managable.
Removed print statements from some tests.
Once we have the new columns in notifications table, the query will need to include the rate multiplier and if the number is international.
The monthly billing query will be built next.
We only need one rate per channel. This reflects that. The provider_rates has been left for now, it is still not being used.
New dao has been added to select the right rate for the given notification_type and date of notificaiton.
- previously this was unbounded, so it got all jobs older then 7 days. In excess of 75,000 🔥
- this meant that the job took (a) a long time and (b) a lot memory and (c) doing the same thing every day
These changes mean that the job has a 2 day eligible window for jobs, minimising the number of eligible jobs in a run, whilst still retaining some leeway in event if it failing one night.
In principle the job runs early morning on a given day. The previous 7 days are left along, and then the previous 2 days worth of files are deleted:
so:
runs on
31st
30,29,28,27,26,25,24 are ignored
23,22 jobs here have files deleted
21 and earlier are ignored.
