S3 metadata only supports ascii characters. Whenever we save data to it
we need to make sure we encode it to save it and then decode it to
display it again to users. This abstraction will act as the place for
that decoding to happen so the rest of the code in our views doesn't
need to care about the encoding abstraction.
From a question on cross-government Slack:
> re the Usage tab - currently it shows 3 financial years - last year,
> this year and next year. is it possible to replace the "next year" tab
> with something more useful? its always going to be blank! I was
> thinking it would be good to have 2 financial years ago, 1 financial
> year ago and this financial year.
This seems like a reasonable idea, and is something we’ve talked about
before. The original intention[1] was that seeing your (unchanged) free
allowance for next year would be useful, but that doesn’t really seem to
be a user need.
***
1. See https://github.com/alphagov/notifications-admin/pull/1094
> so that you can check what your SMS allowance is going to be before
> you actually get into it
The search form makes a post request, so that phone numbers and email
addresses don’t show up in logs or browser history.
At most the API will return 50 results, with some pagination links. We
can’t easily give you links to click in the admin app, because links can
only perform get requests.
Because the value of seeing more than 50 results feels quite low (users
will probably make their search more specific before scrolling through
all 50) let’s just show a message saying only the first 50 results are
displayed.
We were throwing an exception when instantiating a LetterImageTemplate
as we weren't giving it all the arguments it needed.
Now we give it all the correct parameters and add a
test for the method. Ideally we would add a unit test for the flask
route for downloading a letter job CSV (which is currently lacking) but
I did the minimal to be confident I've fixed the bug as I think this
whole code may be fresh for a bit of a rewrite according to Chris.
Original error:
```
File "/Users/davidmcdonald/.virtualenvs/notifications-admin/lib/python3.6/site-packages/notifications_utils/template.py", line 669, in __init__
raise TypeError('image_url is required')
TypeError: image_url is required
```
Some email clients will pre-fetch links in emails to check whether
they’re safe. This has the unfortunate side effect of claiming the token
that’s in the link.
Long term, we don’t want to let the link be used multiple times, because
this reduces how secure it is (eg someone with access to your browser
history could re-use the link even if you’d signed out).
Instead, this commit adds an extra page which is served when the user
clicks the link from the email. This page includes a form which submits
to the actual URL that uses the token, thereby not claiming the token as
soon as the page is loaded.
For convenience, this page also includes some Javascript which clicks
the link on the user’s behalf. If the user has Javascript turned off
they will see the link and can click it themselves. This is going on the
assumption that whatever the email clients are doing when prefetching
the link doesn’t involve running any Javascript.
This Javascript is inlined so that:
- it is run as fast as possible
- it’s more resilient – even if our assets domain is unreachable or the
connection is interrupted, it will still run
We’re going to add an interstitial page that redirects to this new URL.
But we don’t want that redirect to 404 while the change is deploying,
because some boxes will have the new URL and some won’t. So let’s deploy
the new URL to all the boxes first, then the redirect page can safely
take over the new one.
The new URL is going to be `post` not `get` because that feels more
HTTP-y, so we need to make sure that’s part of this change too.
If a service has permission to send international letters then the admin
app should tell template preview, so that template preview knows what
rules to apply when it’s validating the address of the letter.
We don’t need to wait for template preview to start looking at this
query string argument – it will just ignore it for now.
For services with permission, they can now put international addresses
into their spreadsheets without getting a postcode error.
This also means they can start using address line 7 instead of postcode,
since it doesn’t make sense to put a country in a field called
‘postcode’. But this will be undocumented to start with, because we’re
not giving any real users the permission.
It does now mean that the number of possible placeholders (7 + postcode)
is greater than the number of allowed placeholders (7), so we have to
account for that in the one-off address flow where we’re populating the
placeholders automatically. We’re sticking with 6 + postcode here for
backwards compatibility.
Context:
- postal addresses can be made from any of the 7 address lines now, and
the postcode can be in any one of the 7
- we can put errors across a whole row now, not just on individual cells
This commit put errors to do with the postal address as a whole across
the whole row now, rather than tying them to any one cell.
Our rules about address columns are relaxing, so that none of them are
mandatory any more. Instead you just need any 3 of the 7 to make a valid
address.
This commit updates our error messaging to reflect that.
Like we have search by email address or phone number, finding an
individual letter is a common task. At the moment users are having to
click through pages and pages of letters to find the one they’re looking
for.
Users of the API will also be able to search by reference, same as for
emails and text messages. But we only show this hint text to users who
have some API keys.
This commit does two things:
- brings the ‘All organisations’ page back within the platform admin
part of the site (because it’s hard to find otherwise)
- makes the layouts of all the pages within platform admin a bit closer
to the service-specific pages in terms of heading sizes, spacing, etc
so that moving between them doesn’t feel so jumpy
This involves three changes which broke our code.
To validate email addresses, the optional dependency `email-validator`
must be installed<sup>1</sup>. But since we don’t use WTForms’ email
validation, we shouldn’t need to subclass it – it can just be its own
self contained thing. Then we don’t need to add the extra dependency.
When rendering textareas, and extra `\r\n` is inserted at the beginning
<sup>2</sup>. Browsers will strip this when displaying the textbox and
submitting the form, but some of our tests need updating to account for
this.
The error message for when you don’t choose an option from some radio
buttons has now changed. Rather than just accepting WTForms’ new
message, this commit makes the error messages like the examples from
the Design System<sup>3</sup>. By default it will say ‘Select an
option’, but by passing in an extra parameter (`thing`) it can be
customised to be more specific, for example ‘Select a type of
organisation’.
***
1. https://github.com/wtforms/wtforms/pull/429
2. https://github.com/wtforms/wtforms/issues/238
3. https://design-system.service.gov.uk/components/radios/#error-messages
When using `with pytest.raises...` any assertions inside the `with`
statement that occur below the line that raises the exception don't get
called. It's not possible to check the response status_code / location
in this test because an exception is raised before the response is
returned.
A user can't be archived if they are the only member of their service
with `manage_settings` permission. `notifications-api` returns a `400`
and an error message if that is the case, however this PR to remove the
`400` error handler
https://github.com/alphagov/notifications-admin/pull/3320 stopped the
error message from showing. This meant that instead of seeing a message
about why a user couldn't be archived, we would just show a `500` error
page instead. This change checks the response from `notifications-api`
and shows an error banner with a message if the user can't be archived.
I noticed when using the dication software that saying ‘one two three
four five’ got dictated as `123 45`. This tripped the validation,
because the space character isn’t a digit.
So this commit normalises out spaces (and other spacing characters like
dashes and underscores) before validating the code and sending it to the
API.
I can also imagine that some people might like to space out the code to
make it easier to transcribe (like you might do with a credit card
number).
Errors with messages being too long or empty aren’t specific to a single
cell of the uploaded spreadsheet, they’re the results of combining all
the cells with the template.
Previously we could only show errors against a specific cell. This
commit makes it possible to add a super-row which spans all the cells,
into which we can put errors.
The index (header) column then spans both these rows, to show that they
are both associated with the same row of input.
Depends on:
- [x] https://github.com/alphagov/notifications-utils/pull/719
Tests that just assert some content `in` the whole page are tricky to
debug, and make it harder to be sure that said content is showing up in
the right place, with the right markup and styling.
At the moment the first AJAX call is triggered as soon as the page
loads. We then look at its response time to work out how long to wait
until making the next call.
This isn’t great because:
- stuff is unlikely to have changed straight away, so it’s a waste of a
call
- while the DOM is being updated it seems to introduce a delay in
clicks on links, which is either more pronounced or more noticeable
when it’s happening straight away, making the UI feel less snappy
I chose a value of 2 seconds as a rough proxy for the minimum time we’d
expect to see a notification go from created to delivered. Median
time-to-delivered was 2.9 seconds when we analysed it for
https://github.com/alphagov/notifications-admin/pull/2974#discussion_r286101286
If you’re on a page with a normal placeholder and the previous
placeholder is one that’s also in the address block then going back to
the previous page will send you immediately forward to the current page
again.
This commit makes the back link a bit smarter by skipping over pages
where it can see that they relate to a placeholder from the address
block.
If it gets all the way to the start of the list of placeholders without
finding any non-address ones then it will default to generating a link
that will redirect the user to filling in the address block again.
We don’t really want you modifying lines of the address after you’ve
entered it. Especially when it might not be obvious that modifying the
address line placeholder will modify the address you’re sending the
letter to.
Optional address placeholders aren’t a thing for one-off letters any
more, so we can tidy up the code a bit by removing the parts of the flow
that are accounting for them.
If you have an placeholder from the address block elsewhere in your
letter then you currently get redirected to the address block page
instead of being offered to fill that placeholder in. This commit
tightens up the check to only do this when the placeholder is in the
first 7 placeholders, which is where we store the address placeholders.
`dir(object)` is a useful Python function that tells you what attributes
and methods an object has. It’s also used by tools like iPython and IDEs
for code completion.
Some of the attributes of a `JSONModel` are dynamic, based on what
fields we expect in the underlying JSON. Therefore they don’t
automatically end up in the result of calling `dir`. To get around this
we can implement our own `__dir__` method, which also returns the names
of the fields we’re expecting the the JSON.
Inspired by this Raymond Hettinger tweet:
> #python tip: If you add attributes to an API with __getattr__() or
> __getattribute__(), remember to update __dir__() to make the extension
> introspectable.
— https://twitter.com/raymondh/status/1249860863525146624
By default our AJAX calls were 2 seconds. Then they were 5 seconds
because someone reckoned 2 seconds was putting too much load on the
system. Then we made them 10 seconds while we were having an incident.
Then we made them 20 seconds for the heaviest pages, but back to 5
seconds or 2 seconds for the rest of the pages.
This is not a good situation because:
- it slows all services down equally, no matter how much traffic they
have, or which features they have switched on
- it slows everything down by the same amount, no matter how much load
the platform is under
- the values are set based on our worst performance, until we manually
remember to switch them back
- we spend time during incidents deploying changes to slow down the
dashboard refresh time because it’s a nothing-to-lose change that
might relieve some symptoms, when we could be spending time digging
into the underlying cause
This pull request makes the Javascript smarter about how long it waits
until it makes another AJAX call. It bases the delay on how long the
server takes to respond (as a proxy for how much load the server is
under).
It’s based on the square root of the response time, so is more sensitive
to slow downs early on, and less sensitive to slow downs later on. This
helps us give a more pronounced difference in delay between an AJAX call
that is fast (for example the page for a single notification) and one
that is slow (for example a dashboard for a service with lots of
traffic).
*Some examples of what this would mean for various pages*
Page | Response time | Wait until next AJAX call
---|---|---
Check a reply to address | 130ms | 1,850ms
Brand new service dashboard | 229ms | 2,783ms
HM Passport Office dashboard | 634ms | 5,294ms
NHS Coronavirus Service dashboard | 779ms | 5,977ms
_Example of the kind of slowness we’ve seen during an incident_ | 6,000ms | 18,364ms
GOV.UK email dashboard | `HTTP 504` | 😬
We had a report that when clicking on the 'Download this letter' link on
the notification page the file was not being downloaded as a PDF file
but was given a `.htm` file extension instead. We should be able to stop
that happening by using Flask's `send_file` function with the right mimetype.
This change updates the `view_letter_notification_as_preview` to use
`send_file` and splits out code to get the file data into a separate
function.
Mocks in the tests have been updated and some unused mocks removed.
As part of making the API call we extra the recipient from the first
line of the address. This code was assuming that the recipient would
always have the key `address line 1`, but we’re no longer guaranteeing
that it will be capitalised and spaced exactly like that.
