Sometimes a job finishes but has missed a row in the middle. It is a mystery why this is happening, it could be that the task to save the notifications has been dropped.
So until we solve the missing let's find missing rows and process them.
A new scheduled task has been added to find any "finished" jobs that do not have enough notifications created. If there are missing notifications the job processes those rows for the job.
Adding the new task to beat schedule will be done in the next commit.
A unique key constraint has been added to Notifications to ensure that the row is not added twice. Any index or constraint can affect performance, but this unique constraint should not affect it enough for us to notice.
Since Pytest 5, `ExceptionInfo` objects (returned by `pytest.raises`) now
have the same `str` representation as `repr`. This means that `str(e)`
now needs to be changed to `str(e.value)`.
https://github.com/pytest-dev/pytest/issues/5412
The nightly job to delete email notifications was failing because it was
timing out (`psycopg2.errors.QueryCanceled: canceling statement due to statement timeout`).
This adds a query limit to the query which inserts or updates
notification history so that it only updates a maximum of 10000 rows at
a time.
All our endpoint should perform a check that the params are valid - this is an easy whay to check that and is standard for our endpoints.
I reverted the query to just filter by job id.
When a service asks for branding I often go and:
- set the branding on the organisation as well
- set the branding on all the organisation’s services
The latter can be quite time consuming, but it does save effort since
existing services from the same organisation won’t have to request
branding. It also improves the consistency of the communications that
users are receiving.
This commit automates that process by applying the branding update to
any services belonging to the organisation, if those services don’t
already have their own custom branding set up.
Now we consistently use the created_at date, so we can always get the right file location and name.
The previous updates to this code were trying to solve the problem if a pdf being created at 17:29, but not ready to upload until 17:31 after the antivirus and validation check.
But in those cases we would have trouble finding the file.
When we cancel a job, we need to check if all notifications are
already in the database. So far, we were querying for all
notification objects in the database and counting them in
admin app, which runs into pagination problems for large jobs,
and could time out for very large jobs.
Code that is within a `with Python.raises(...)` context manager but
comes after the line that raises the exception doesn't get evaluated.
We had some assertions that we never being tested because of this, so
this ensures that they will always get run and fixes them where
necessary.
bst_date is a date field. Comparing dates with datetimes in postgres
gets confusing and dangerous. See this example, where a date evaluates
as older than midnight that same day.
```
notification_api=# select '2019-04-01' >= '2019-04-01 00:00';
?column?
----------
f
(1 row)
```
By only using dates everywhere, we reduce the chance of these bugs
happening
Previously we were doing it based on their email address. This will also
apply it if they self-select as a GP surgery, even if they don’t have an
NHS email address.
This is the second commit in the series to add organisation_id to Service.
- Data migration to update services.organisation_id from data in organisation_to_service
(The rollback will lose any updates to organisation unless the script is updated to set organistion_to_service from service.organisation_id )
- Update Service.organisation relationship to a ForeignKey relationship to Organisation.
- Update Organisation.services to a backref relationship to Service.
- Add oranisation_id to Service data model.
- Update methods to create service and associate service to organisation to set the organisation_id on the Service.
- Create the missing test, if the service user email matches a domain for an organisation then associate the service to the organisation and inherit crown and organisation_type from the organisation.
For example if a file is uploaded on Aug 1, but scheduled for Aug 3, the csv file will be deleted 2 days before the notifications.
This will cause an error for on the jobs pages if the download report link is clicked.
This content was first done without help of content designer so
now it is improved with the help of one. New content is more
in line with the content in the admin app and more human-readable
and following GOV.UK standard.
This PR updates those queries to look in either Notification or NotificationHistory. Since the data does not exist in both tables at the same time we can do with and not worry about the data retention.
The query will iterate over each service, then each notification type and query the data if no results then try the history table.
The organisation_type of a service should match the organisation_type of
the service's organisation (if there is one). This changes
dao_update_organisation and dao_add_service_to_organisation to set the
organisation_type of any services when adding / updating an org.
test can_letter_job_be_cancelled closer to the code
test dao_cancel_letter_job closer to the code
Mock out calls in cancel_letter_job to test just that method
