Separated s3_client.py into 3 files - for logos, CSV files and the MOU.
This helps to keep things clearer now that we need to add lots more logo
functions for letters.
We have some teams who haver a series of files they have to send each
day. It’s easy to get muddled up and accidentally send the same file
again, if you think you haven’t already sent it.
This commit blocks you from sending the same combination of template
version and filename more than once on the same day[1].
This won’t affect teams who re-use the same template to give (for
example) updates on an incident for business continuity. These teams
edit the template between each send, thereby updating the version
number of the template.
1. This is based on how the `limit_days` argument to the API works - you
can dig into the code here: 2bd4f74ad0/app/dao/jobs_dao.py (L50)
Because we’re going to use a sticky footer on this page we don’t need
to show the email in its collapsed form. Showing the email is only
needed to:
- fit more things on a page
- make sure a button comes up above the fold (which the sticky footer
will solve)
Currently, a user can select a reply-to email address or text message
sender when uploading a CSV file but this is ignored and the default is
always used instead. As a first step towards changing this, this adds
the sender_id (if selected) to the S3 metadata so that this information
can be used when processing the job.
We didn’t used to allow this because it wasn’t really possible with the
old DVLA set up and we didn’t think there’s a need.
We think it’s possible now because, even though it’s cumbersome, it’s
better than the manual process.
There’s a lot of code in service settings which:
- talks to the API directly through the clients
- passes that information through to the Jinja template
By encapsulating this logic in the service model:
- the Jinja template can access the data directly
- the logic can be reused across multiple methods
We’ve had a support ticket saying:
> Hi, where a letter goes over to two sides, is there a way in the
> 'Preview' screen (or anywhere else) that I can see page two? I can
> see page one OK, but can't work out how to see what's generated on the
> second page.
Whether you’re about to send 1000s of letters – or just want to preview
how one will look – it’s probably useful to be able to see more than
just the first page.
Admin, API and utils were all defining a value for SMS_CHAR_COUNT_LIMIT.
This value has been updated in notifications-utils to allow text
messages to be 4 fragments long and notifications-admin now gets the value of
SMS_CHAR_COUNT_LIMIT from notifications-utils instead of defining it in
config.
If you skip past the templates page (because you don’t have the edit
permission) but then click back you end up in a loop which redirects you
to the page you’re already on.
This commit makes sure that you’re sent back a step further, so you
don’t get stuck in that loop.
The one downside of skipping the template page is that you no longer
get such strong confirmation that you’ve picked the correct template.
You still see the preview of the template, but it’s further down the
page, and the name of the template has disappeared.
This commit adds the name of the template to the page title, to:
- have some continuity from the previous page
- make it easier to double-check you’ve chosen the correct template
‘Upload recipients’ and ‘Send to one recipient’ have always been
slightly clunky phrases.
Now that basic view jumps straight into the ‘Send to one recipient’
flow there’s no way for users to get to the ‘Upload recipients’ flow.
By adding a link to it from the ‘Send to one recipient’ flow it’s
possible for users of basic view to access it.
But we don’t want to introduce too much inconsistency between basic view
and admin view because users will be migrating from one to another. They
might also be talking to their manager, who wouldn’t be able to tell
them where to click if they were looking at two completely different
interfaces.
This also means that we can keep the left-hand navigation in basic view
nice and simple with the two options (‘Templates’ and ‘Sent messages’),
rather than trying to introduce something like ‘Send one message’ and
‘Send lots of messages’ later on.
This is better than just keying into the JSON because it means you get
an exception straight away when looking up a key that doesn’t exist
(which via mocking you could ordinarily miss).
Having the service floating about as JSON is a bit flakey. Could easily
introduce a mistake where you mistype the name of a key and silently
get `None`.
Also means doing awkward things like `if 'permission' in
current_service['permissions']`, whereas for users we can do the
much cleaner `user.has_permission()`.
So this commit:
- introduces a model
- adds a `.has_permission` method similar to the one we have for users
Commit 58cc1604a7 sanitises any non-ascii
characters in the headers. CSV filenames get used as a header value, so
this fixed a bug that occurred when non-ascii characters were used.
The CSV filename also gets used as part of the metadata when uploading
the file to S3. Since the S3 metadata can only contain ASII characters,
we also need to sanitise the filename before uploading it to S3.
This link is useful for people who are setting up templates and want to
test out how they look/how Notify works.
‘Caseworker’ users shouldn’t need to send themselves messages on a
regular basis, so this link is another thing we can take away.
Caseworkers skip the template page in their message sending journey.
Instead they go straight from picking a template to the first step of
sending. So the ‘Back’ link should send them straight back to the
picking a template page, skipping the individual template page (which
they don’t have permission to view).
If a template has a placeholder like `((email address))` then the sample
spreadsheet and CSV file have the email column twice.
Trying to upload this spreadsheet will result in a ‘duplicate column’
error.
This commit fixes it so that the column will only appear once.
We want to set the metadata on a file when rendering the check page, but
not when rendering the preview of a letter. So let’s put the code that
does that only in the place where we’re rendering the check page. This
means not having to pass a variable around.
Refactoring only; no functional changes.
When you upload a CSV the check page takes the file name from the URL
and writes it to the S3 metadata for that file.
This also happens when you go to the .png version of this page.
The .png version of the page doesn’t have the filename in the URL. So it
re-writes the S3 metadata with an empty filename.
This means that all letter jobs sent recently have no file names. This
commit fixes this problem.
we're not actually looking at the detailed service aspects - just
the stats. We're doing this in three places:
* dashboard
* notification activity page
* when checking jobs to see if we're over the daily limit
change these places to use a new api endpoint (service/id/statistics),
which hopefully be a little more performant, and will definitely be a
little more organised - moving away from generic endpoints with loads
of optional parameters.
We still need the detailed endpoints for the platform admin page tho.
Depends on https://github.com/alphagov/notifications-api/pull/1865
If a user clicks ‘back’ once they’ve sent a job we don’t want them to
land on the ‘check’ page again. This would suggest that they can send
the same job again (they can’t because that `job_id` is in the database
already). That said, it’s confusing to see that page; the natural thing
is to go jump back another step, to where you uploaded the file.
We’re going to stop storing job metadata in the session. So we can’t
rely on it for checking whether a file is valid. That safeguard is
happening in the API instead now (because it’s looking at the metadata
stored in S3).
S3 has a limit of 2kb for metadata:
> the user-defined metadata is limited to 2 KB in size. The size of
> user-defined metadata is measured by taking the sum of the number of
> bytes in the UTF-8 encoding of each key and value.
– https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata
This means we have a limit of 1870 bytes for the filename:
```python
encoded = 'notification_count50000template_id665d26e7-ceac-4cc5-82ed-63d773d21561validTrueoriginal_file_name'.encode('utf-8')
sys.getsizeof(b)
>>> 130
2000-130
>>> 1870
```
Or, in other words, ~918 characters:
```python
sys.getsizeof(('ü'*918).encode('utf-8'))
>>> 1869
```
We’re getting the template just to get back its `id`, which is the one
thing we do know in order to get it.
The call to get template is still happening inside `_check_messages`, so
we’ll still catch someone trying to look at this page for a template
that doesn’t exist.
By doing this we no longer have to store it in the session. This is the
last thing that’s currently in the session, so removing it means we can
drop session storage for file uploads entirely.
Storing things in the session is proving buggy – we still have one user
(that we know about) where the session data isn’t getting written, so
they’re blocked from uploading a file.
Since all the info we’re storing in the session is about the file, it
makes sense to store it with the file.
This commit only does the writing of the metadata, once we’re sure this
is working we can do subsequent work to read it back, and remove
reliance on the session.
The check page expects template ID to be passed through in the URL not
the session now. The send test letter page wasn’t changed.
This commit changes it, and adds a test to make sure this path is
covered.
The start job endpoint needs the template ID in order to make the API
call.
It doesn’t make sense to add it to the start job URL, because users
could potentially start a job with the wrong template by hacking the URL
(which would blow up at some point, if the template didn’t match the
columns in the file).
A of this commit’s parent we are storing `template_id` and
`original_file_name` in the URL. Getting them from the URL is better,
so the check page no longer needs to look for them in the session. This
commit removes the code that looks for these values in the session.
At the moment you can’t press refresh on the check page if there’s
errors. This is because the session gets cleared when there’s errors.
This is a bad user experience.
The data that this page is relying on (from the session) is:
- template ID
- original file name
Neither of these things need to be in the session because:
- they are not secret
- the user can modify them already (by choosing a different template or
renaming their file locally)
So this commit additionally stores them in the URL.
if a user signs in again, clear their file upload data from any
aborted journeys from before, so that their cookies don't fill up
also add some temporary logging when the session starts getting full.
Because we now[1] store info about each file upload separately in the
session the session isn’t overridden every time you upload a file. This
is good because you can do multiple file uploads idempotently.
Generally we are cleaning up after ourselves because we pop anything to
do with that upload from the session. However there is an edge case: if
you never send the file then the info about the file stays in the
session in perpetuity[2]. This is generally happening when people are
uploading files that are impossible to send, ie ones that have errors.
So this commit makes two changes:
1. remove info about a file upload from the session as soon as we know
that it contains errors
2. `POST` reuploads to the same endpoint as initial uploads because
otherwise we need to keep info about bad uploads in the session,
which would prevent us from doing 1.
1. https://github.com/alphagov/notifications-admin/pull/1968
2. or at least until the session is cleared by the user logging out
If you have a placeholder called `((phone number))` in your email
template, and you try to send a one-off message then the form input will
attempt to validate your ‘phone number’.
This is not helpful if you’re trying to put a landline number in your
email, for example.
This only affects messages being sent through the one-off interface.
This commit makes the form be aware of template type, which fixes the
problem.
