The nightly job to delete email notifications was failing because it was
timing out (`psycopg2.errors.QueryCanceled: canceling statement due to statement timeout`).
This adds a query limit to the query which inserts or updates
notification history so that it only updates a maximum of 10000 rows at
a time.
All our endpoint should perform a check that the params are valid - this is an easy whay to check that and is standard for our endpoints.
I reverted the query to just filter by job id.
When a service asks for branding I often go and:
- set the branding on the organisation as well
- set the branding on all the organisation’s services
The latter can be quite time consuming, but it does save effort since
existing services from the same organisation won’t have to request
branding. It also improves the consistency of the communications that
users are receiving.
This commit automates that process by applying the branding update to
any services belonging to the organisation, if those services don’t
already have their own custom branding set up.
Now we consistently use the created_at date, so we can always get the right file location and name.
The previous updates to this code were trying to solve the problem if a pdf being created at 17:29, but not ready to upload until 17:31 after the antivirus and validation check.
But in those cases we would have trouble finding the file.
When we cancel a job, we need to check if all notifications are
already in the database. So far, we were querying for all
notification objects in the database and counting them in
admin app, which runs into pagination problems for large jobs,
and could time out for very large jobs.
Code that is within a `with Python.raises(...)` context manager but
comes after the line that raises the exception doesn't get evaluated.
We had some assertions that we never being tested because of this, so
this ensures that they will always get run and fixes them where
necessary.
also create a PDFNotReadyError class, separate to BadRequestError, to
imply to the end user that this is something they should handle
separately to all the other errors
it felt very awkward when the body of a pdf might be empty, might have
things in it, and whether it is empty or not can change even when the
status is the same (a created template notification might have a pdf,
but might not, we don't know).
So move it to its own endpoint, so we can hand craft some 400 errors
that appropriately explain what's going on.
The `get_bucket_name_and_prefix_for_notification` function was looking
at the `sent_at` or `updated_at` at time of a notification to see which
bucket it was in. Precompiled letters sent through the admin app don't
have either of these times - they only have a `created_at` time, so this
lets the function check `created_at` time too.
This function checks various permissions, downloads the PDF from the
transient bucket, creates the notification then moves the letter to the
'normal' bucket.
This has been moved to the letters utils file since it will be used in
more than one place. The notification parameter has been removed so that
the function can be used when we don't have a notification id.
We want a way of getting the hidden precompiled template from admin,
so this adds an endpoint which gets the template or creates it if it doesn't
exist. The function to get or create the hidden template already existed
but has been moved to the template DAO now that it is used in more
places.
previously, we didn't create templated letters, and just marked them as
delivered straight away. However, we may need to return PDFs for these
letters, so we should create them the same as live letters. Then update
the functions so that they know where to look for these letters.
this is only applicable when getting a single notification by id. it's
also ignored if the notification isn't a letter.
Otherwise, it overwrites the 'body' field in the response.
If the notification's pdf is available, it returns that, base64
encoded. If the pdf is not available, it returns an empty string.
The pdf won't be available if the notification's status is:
* pending-virus-scan
* virus-scan-failed
* validation-failed
* technical-failure
The pdf will be retrieved from the correct s3 bucket based on its type
This will let us do some filtering of this list in the admin. It’s
better to do it there because it means the admin can use the same cached
response from Redis each time.
Using the created at date for the folder is not always going to work because the pdf created_at date could be just before the cut off date but virus scan and validation has yet to happen. By the time the letters is in the created state, the letter goes into the next days bucket.
It can also happen if the letters is stuck in `pending-virus-scan` and we need to restart the task, and the letters is in a different folder.
bst_date is a date field. Comparing dates with datetimes in postgres
gets confusing and dangerous. See this example, where a date evaluates
as older than midnight that same day.
```
notification_api=# select '2019-04-01' >= '2019-04-01 00:00';
?column?
----------
f
(1 row)
```
By only using dates everywhere, we reduce the chance of these bugs
happening
