The only impactful change is the major version itself, where I've
fixed the breaking changes due to the upgrade of PyPDF2 [^1] and
checked there are no deprecation warnings when I run the tests.
[^1]: https://github.com/alphagov/notifications-utils/pull/973
Previously we weren't sure if the cause of this exception was what
the comment below suggested [1]. I've now verified this from:
Letter not found for service 0bd1d970-f11c-40e1-8319-4baefe6239d7 and file aa07ed06-3161-4795-93b3-b45d7c576af9
I checked that aa07ed06-3161-4795-93b3-b45d7c576af9 exists already
as a notification i.e. the comment is correct and we're not sending
users to a 404 page. It's possible there are other scenarios where
the comment is wrong, but I don't think it's worth keeping the error.
[1]: https://github.com/alphagov/notifications-admin/pull/4159
This repeats the pattern we already have for previewing a letter,
where we assume the error is because the notification has already
been sent and redirect the user to see it.
I've improved the original pattern a bit:
- I've DRYed-up the low-level boto code and moved the error handler
there so it can be reused.
- I've introduced a custom exception, which the calling code can
choose to log.
- I've introduced the moto library, which we use elsewhere, to make
it easier to test S3 code.
I've used an error level log when sending a notification - now that
we have a more descriptive log, we can verify the assumption is true
and then make an informed decision to downgrade the log.
In future we may want to merge this handler with the similar code
in utils [1], but we'll need to be careful as the utils handler is
superficial - it doesn't check the reason for the error.
[1]: bce0f4e596/notifications_utils/s3.py (L52)
Changes:
53.0.0
---
* `notifications_utils.columns.Columns` has moved to
`notifications_utils.insensitive_dict.InsensitiveDict`
* `notifications_utils.columns.Rows` has moved to
`notifications_utils.recipients.Rows`
* `notifications_utils.columns.Cell` has moved to
`notifications_utils.recipients.Cell`
52.0.0
---
* Deprecate the following unused `redis_client` functions:
- `redis_client.increment_hash_value`
- `redis_client.decrement_hash_value`
- `redis_client.get_all_from_hash`
- `redis_client.set_hash_and_expire`
- `redis_client.expire`
51.3.1
---
* Bump govuk-bank-holidays to cache holidays for next year.
51.3.0
---
* Log exception and stacktrace when Celery tasks fail.
This brings a few performance improvements for RecipientCSV, which
we use to preview and process CSVs. One change also renames one of
the attributes for the class to "guestlist".
This has several advantages:
- It gives us more room to explain the error and actions. This will
be useful for upcoming work we want to do, which will add yet more
validations for CSV uploads.
- We already use a flash to show certain kinds of errors on these
pages (just above). This is more consistent.
- It's potentially more accessible. Previously the error and the
button text used to be read out as a single sentence. Now the page
reloads and reads the flash error alone.
In theory we should show an error in both places, but this can be
confusing on pages where there's only a single form control, and
especially if the error is long.
This continues the work from Template Preview [1], so that we have
a complete store of original PDFs to use for testing changes to it.
Previously we did store some originals, but these were only invalid
PDFs that had failed sanitisation; for valid PDFs, the "transient"
bucket only contains the sanitised versions, which the API deletes
/ moves when the notification is sent [2].
Since the notification is only created at a later stage [3], there's
no easy way to get the final name of the PDF we send to DVLA. Instead,
we use the "upload_id", which eventually becomes the notification ID
[4]. This should be enough to trace the file for specific debugging.
Note that we only want to store original PDFs if they're valid (and
virus free!), since there's no point testing changes with bad data.
[1]: https://github.com/alphagov/notifications-template-preview/pull/545
[2]: c44ec57c17/app/service/send_notification.py (L212)
[3]: 7930a53a58/app/main/views/uploads.py (L362)
[4]: 7930a53a58/app/main/views/uploads.py (L373)
On the uploads page we only show jobs which are within a service’s data
retention.
This commit does the same for when we’re listing the jobs for a contact
list. This matches the UI, which says a contact list has been ‘used
`<count_of_jobs>` in the last <data_retention> days’
It’s a bit unintuitive that starting a job from a contact list makes a
copy of the file, which has no relationship to the list it was copied
from. This is more of an implementation detail, rather than something
that comes from people’s mental models of what is going on. Or at least
that’s what I hypothesise.
I think it’s clearer to show jobs that come from contact lists within
the lists that they were created from. By naming the jobs by template
this gives a clearer view of what messages have been sent to the group
over time.
The code was looking for `original_file_name` in the metadata for a
contact list, or the query string if it wasn't in the metadata. Now that
the change to use the metadata for the file name has been deployed for a
while e can stop looking in the query string for the
`original_file_name`.
We were passing `original_file_name` from the `.upload_contact_list`
view function to the `.check_contact_list` view function as a query
param. We now store it in the metadata instead. `.check_contact_list`
still checks for `original_file_name` in the query string if it's not in
the metadata - this is necessary until the code has been deployed for a
few days and we can be sure that there are no contact lists that are
mid-way through the upload stage.
the upload preview page has a file_id - this corresponds to the file in
the transient pdf uploads bucket. However, if the user already hit send
(and then navigated back) the file's no longer in that bcuket, it's been
moved to the regular letters-pdf bucket. So the s3 get request fails. To
avoid this, simply redirect to the notifications page if the file isn't
in the transient bucket. This is better for the user as it'll stop them
trying to submit it twice, and will provide more clarity on the status
of the notification too.
International letters don’t have a choice of postage. Under the hood
they are either `europe` or `rest-of-world`.
So, for letters that we detect are international, this commit:
- removes the radios buttons that give users the choice of postage
- passes through either `europe` or `rest-of-world` to the API,
depending on what address we find in the letter
This will cause the API to 500 until it can accept `europe` or
`rest-of-world` as postage types, but this is probably OK because it’s
only our services that have international letters switched on at the
moment.
Because we no longer need the form to get the `file_id`, we can get the
metadata before building the form.
This will, in subsequent commits, let us build the form differently
based on the recipient metadata.
Also removed some variables that were assigned to then only used once
and reformatted arguments for readability.
In the future we need to get the metadata from the file in order to work
out what form validation rules should apply (postage is only required
for UK letters).
To start doing this we need all instances of the app accepting `post`
requests with the `file_id` in the URL, as well as in the form data (for
backwards compatibility).
This is for consistency with how we do it for filenames in the previous
commit and moves the decoding into the `LetterMetadata` class for
abstracting this behaviour.
Small refactor of the LetterMetadata class needed to handle None case as
recipient can be None.
If you’ve come to look at a notification via the uploaded letters page
then the ‘< back’ link should take you back there, not to the usual
activity page.
Some teams have started uploading quite a lot of letters (in the
hundreds per week). They’re also uploading CSVs of emails. This means
the uploads page ends up quite jumbled.
This is because:
- there’s just a lot of items to scan through
- conceptually it’s a bit odd to have batches of things displayed
alongside individual things on the same page
So instead we’re going to start grouping together uploaded letters. This
will be by the date on which we ‘start’ printing them, or in other
words the time at which they can no longer be cancelled.
This feels like a natural grouping, and it matches what we know about
people’s mental models of ‘batches’ and ‘runs’ when talking about
printing.
This grouping will be done in the API, so all this commit need to do is:
- be ready to display this new type of pseudo-job
- link to the page that displays all the uploaded letters for a given
print day
Because we won’t be showing uploaded letters individually on the uploads
page any more we need a way of listing them. This should be by printing
day, to match how we’re grouping them on the uploads page.
This code reuses the notifications.html template, but flips the
precedence of the filename and recipient because I reckon when you’re
looking at uploads you’re thinking filename-first.
If a service has permission to send international letters then the admin
app should tell template preview, so that template preview knows what
rules to apply when it’s validating the address of the letter.
We don’t need to wait for template preview to start looking at this
query string argument – it will just ignore it for now.
For services with permission, they can now put international addresses
into their spreadsheets without getting a postcode error.
This also means they can start using address line 7 instead of postcode,
since it doesn’t make sense to put a country in a field called
‘postcode’. But this will be undocumented to start with, because we’re
not giving any real users the permission.
It does now mean that the number of possible placeholders (7 + postcode)
is greater than the number of allowed placeholders (7), so we have to
account for that in the one-off address flow where we’re populating the
placeholders automatically. We’re sticking with 6 + postcode here for
backwards compatibility.
Our rules about address columns are relaxing, so that none of them are
mandatory any more. Instead you just need any 3 of the 7 to make a valid
address.
This commit updates our error messaging to reflect that.
The saved uploads page just shows you the same thing you’ve seen on the
previous page – a list of email addresses.
It’s more useful to see the uploads page, so you can see where the thing
you’ve saved has gone. You can see it more in context.
Uploads page is where all the stuff you’ve uploaded lives. Now you can
upload contact lists they should live here too.
They always come first because they’re the most-removed from stuff
you’ve sent.
