Which means the sent_at date for the notification could be empty causing the service callback to fail.
- Allow code to work if notification.sent_at or updated_at is None
- Update calls to send_delivery_status_to_service to send the data encrypted so that the task does not need to use the db.
When the notification is timedout by the scheduled task if the service is expecting a status update, that update to the service would fail.
A test has been added.
The tasks are no longer being used, so can be deleted safely:
* record_initial_job_statistics
* record_outcome_job_statistics
* timeout-job-statistics
The test file for the statistics tasks was deleted in a previous commit.
the process_incomplete_jobs task runs through all incomplete jobs in
a loop, so it might not get a chance to update the processing_started
time of the last job before check_job_status runs again (every minute).
So before we even trigger the process_incomplete_jobs task, lets set
the status of the jobs to error, so that we don't identify them for
re-processing again.
we might stop processing jobs mid-way through if, for example, a
deploy or downscale kills the box working on it. We have a scheduled
task that identifies any job that we started processing more than half
an hour ago that is still processing.
However, we encountered a bug where we triggered the
process_incomplete_job multiple times, because the processing_started
of the job was still set to half an hour ago. If we reset the
processing_started to the current time, then it won't get picked up by
future runs of the check_job_status scheduled task.
We don’t have any way of playing back the totals we send to performance
platform.
This commit copies the command used to backfill the processing time and
adapts it to backfill the totals instead. Under the hood it uses the
same code that we use in the scheduled tasks to update performance
platform on a daily basis. I had to modify this code to take a `day`
argument because it was hardcoded to only work for ‘yesterday’.
- Also convert the files info to upper() for comparison rather than lower
because original file names are in upper case. The unit tests contain examples of the returned lists.
Since preview and staging environments don't have a full DVLA
integration they're likely to contain letter notifications in
a 'sending' state. To avoid spamming Deskpro we skip the check
unless we're in a production or test environment.
We should receive a response file from DVLA by 4pm the next working
day (next Monday for letters created on Friday, Saturday or Sunday).
Response file triggers a task to update the letters status from
'sending' to either 'failed' or 'delivered', at which point there
should be no letter notifications in the 'sending' state for that day.
To catch any errors in the process (eg a missing response file from
DVLA) we add a scheduled task that checks letter notifications for
previous day (or Friday when run on Monday) and raises a Deskpro
ticket if it finds any in a 'sending' state. We're checking letter
notifications based on the `sent_at` date, which is set when the
letter PDF is sent to DVLA (so for letters created after 5:30pm it
will be the next day).
The task runs at 4:30pm, which should give the response file processing
task enough time to finish if the file was uploaded at 4pm.
`create_custom_template` is not using `dao_create_template` since
it explicitly sets a template id. Notifications.template relationship
now refers to a TemplateHistory objects, so we need to make sure that
`create_custom_template` creates a matching TemplateHistory record
when it creates a Template.
There are some Null template_ids in the production database which was
causing a failure as the stats_template_usage_by_month has a constraint
that it should not be null. This update only adds populated template_ids
- Updated Celery task
- Add a new test to test for null template_ids and not try to add them
to the stats
- Modified the services_dao to return an int instead of a datetime to
make usage easier and removed the BST function on year as it is not
relevant for year
- Improved tests do there is less logic by ordering the result so there
is less reliance on the template id
- Renamed variable in stats_template_usage_by_month_dao.py to make it
consistent with the method
- Some tests for failing because of an import which was overriding the
one above it in test_scheduled_tasks.py
- Import error fixed in test_services_dao.py after a rename of a method
Currently some pages are timing out due to the time it takes to perform
database queries. This is an attempt to improve the performance by
performing the query against the notification history table once a day
and use the notification table for a delta between midnight and the when
the page is run and combine the results.
- Added Celery task for doing the work
- Added a dao to handle the insert and update of the stats table
- Updated tests to test the new functionality
- Moved the process_incomplete_jobs to tasks.py
- Moved the process_incomplete_jobs test to test_tasks.py
- Cleaned up imports and other code style issues.
As the new tasks is not a scheduled one, moved the the tasks.py file.
This makes it more consisted with other tasks. Updated a few code style
issues to make it more consistent with other coe and hence more
maintainable in future.
- Added a new task to process incomplete jobs
- Added tests to test the new method
- Updated the check for incomplete jobs method to start the new task
This will effectively resume tasks which for some reason were interrupted
whilst they were being processed. In some cases only some of the csv
was processed, this will find the place in the csv and continue processing
from that point.
- Added code in `delete_notifications_created_more_than_a_week_ago_by_type` to remove notifications to email_reply_to older than 7 days
- Added transactional to `delete_notifications_created_more_than_a_week_ago_by_type`
Removed the tests for trial mode service for the scheduled tasks and the process job.
Having the validation in the POST notification and create job endpoint is enough.
Updated the test_service_whitelist test because the order of the array is not gaurenteed.
