If a template has a placeholder like `((email address))` then the sample
spreadsheet and CSV file have the email column twice.
Trying to upload this spreadsheet will result in a ‘duplicate column’
error.
This commit fixes it so that the column will only appear once.
If you’re in trial mode you can’t send letters for real. But you can
still upload a spreadsheet with multiple rows, and there’s no reason why
you shouldn’t be able to explore how Notify populate the letter for each
row of the spreadsheet (since this is something we let you do when you
can send the messages for real).
Updated notifications-utils. This brings in
- the renamed character sanitization classes
- the change to allow unicode in letter addresses (this lets us delete
a test that is no longer relevant)
Also replaced non-ascii characters in headers. This fixes a bug where
non-ascii characters in a CSV filename were causing errors since the
filename is also used in the header.
When you upload a CSV the check page takes the file name from the URL
and writes it to the S3 metadata for that file.
This also happens when you go to the .png version of this page.
The .png version of the page doesn’t have the filename in the URL. So it
re-writes the S3 metadata with an empty filename.
This means that all letter jobs sent recently have no file names. This
commit fixes this problem.
we're not actually looking at the detailed service aspects - just
the stats. We're doing this in three places:
* dashboard
* notification activity page
* when checking jobs to see if we're over the daily limit
change these places to use a new api endpoint (service/id/statistics),
which hopefully be a little more performant, and will definitely be a
little more organised - moving away from generic endpoints with loads
of optional parameters.
We still need the detailed endpoints for the platform admin page tho.
Depends on https://github.com/alphagov/notifications-api/pull/1865
If a user clicks ‘back’ once they’ve sent a job we don’t want them to
land on the ‘check’ page again. This would suggest that they can send
the same job again (they can’t because that `job_id` is in the database
already). That said, it’s confusing to see that page; the natural thing
is to go jump back another step, to where you uploaded the file.
We’re going to stop storing job metadata in the session. So we can’t
rely on it for checking whether a file is valid. That safeguard is
happening in the API instead now (because it’s looking at the metadata
stored in S3).
S3 has a limit of 2kb for metadata:
> the user-defined metadata is limited to 2 KB in size. The size of
> user-defined metadata is measured by taking the sum of the number of
> bytes in the UTF-8 encoding of each key and value.
– https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#object-metadata
This means we have a limit of 1870 bytes for the filename:
```python
encoded = 'notification_count50000template_id665d26e7-ceac-4cc5-82ed-63d773d21561validTrueoriginal_file_name'.encode('utf-8')
sys.getsizeof(b)
>>> 130
2000-130
>>> 1870
```
Or, in other words, ~918 characters:
```python
sys.getsizeof(('ü'*918).encode('utf-8'))
>>> 1869
```
By doing this we no longer have to store it in the session. This is the
last thing that’s currently in the session, so removing it means we can
drop session storage for file uploads entirely.
Storing things in the session is proving buggy – we still have one user
(that we know about) where the session data isn’t getting written, so
they’re blocked from uploading a file.
Since all the info we’re storing in the session is about the file, it
makes sense to store it with the file.
This commit only does the writing of the metadata, once we’re sure this
is working we can do subsequent work to read it back, and remove
reliance on the session.
we were seeing isort produce different outputs locally and in docker -
this was due to it having different opinions about whether the tests
module (ie all our unit tests) is a first party (local) or third party
(pip installed) import. It's a first party import, so by defining this
in the setup.cfg isort settings, we can force it to be consistent
between environments.
Note: I don't know why it was different in the first place though
The check page expects template ID to be passed through in the URL not
the session now. The send test letter page wasn’t changed.
This commit changes it, and adds a test to make sure this path is
covered.
The start job endpoint needs the template ID in order to make the API
call.
It doesn’t make sense to add it to the start job URL, because users
could potentially start a job with the wrong template by hacking the URL
(which would blow up at some point, if the template didn’t match the
columns in the file).
A of this commit’s parent we are storing `template_id` and
`original_file_name` in the URL. Getting them from the URL is better,
so the check page no longer needs to look for them in the session. This
commit removes the code that looks for these values in the session.
At the moment you can’t press refresh on the check page if there’s
errors. This is because the session gets cleared when there’s errors.
This is a bad user experience.
The data that this page is relying on (from the session) is:
- template ID
- original file name
Neither of these things need to be in the session because:
- they are not secret
- the user can modify them already (by choosing a different template or
renaming their file locally)
So this commit additionally stores them in the URL.
Because we now[1] store info about each file upload separately in the
session the session isn’t overridden every time you upload a file. This
is good because you can do multiple file uploads idempotently.
Generally we are cleaning up after ourselves because we pop anything to
do with that upload from the session. However there is an edge case: if
you never send the file then the info about the file stays in the
session in perpetuity[2]. This is generally happening when people are
uploading files that are impossible to send, ie ones that have errors.
So this commit makes two changes:
1. remove info about a file upload from the session as soon as we know
that it contains errors
2. `POST` reuploads to the same endpoint as initial uploads because
otherwise we need to keep info about bad uploads in the session,
which would prevent us from doing 1.
1. https://github.com/alphagov/notifications-admin/pull/1968
2. or at least until the session is cleared by the user logging out
The email template does this already when formatting the body of the
message. But the spreadsheet preview doesn’t, which means you get lists
like:
- thing
- thing
- None
This commit fixes that.
This was a pre-existing bug, but gonna roll it in with this PR.
when added to a service, all users are given the view_activity
permission. So, if that's included in the list, we don't need `any_`,
and we don't need any of the other permissions.
If someone has duplicate recipient columns in their file we don’t know
which one to use. This commit adds an error message which should help
them fix the duplication.
This commit doesn’t go to the extra effort to actually show the
correct values for duplication in the preview. Don’t think it’s worth
the effort/complexity for how infrequently we’ve seen this error.
Depends on:
- [ ] https://github.com/alphagov/notifications-utils/pull/376
Excel stores dates as floating point numbers, counting the days (or
fraction thereof) since 1900 (except when it counts from 1904).
However it also, incorrectly, recognises 1900 as a leap year. This means
that it’s ambiguous whether day 59 is February 28th, or February 27th,
depending if you’re counting up or down. In fact any number less than 60
is, therefore, ambiguous.
This shouldn’t really matter since no-one is going to be using dates in
the year 1900 in Notify messages. _Except_ when Excel misidentifies a
cell as containing a date. For example, if you have the number `9`
inside a cell, it means _house number 9_ if the next cell contains
_example_ street. but Excel is all like ‘oh they must want January 9th
1900!’ No. Bad Excel.
There’s not much we can do about this, but we can at least give people
an error message with the workaround, which is what this commit does.
Most of this commit message is paraphrased from:
http://xlrd.readthedocs.io/en/latest/dates.html
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
Both `<button type='submit'>Submit<button>` and
`<input type='submit' value='Submit'>` can be used to submit a form.
We have historically[1] used `<input>` because it’s better-supported by
IE6 in that:
- the `submit` attribute is mandatory on `<button>`, not on `<input>`
- the `innerHTML` of a button will be submitted to the server, not the
value (as in other browsers)
Reasons to now use `<button>` instead:
- IE6/7 support is no longer a concern (especially with deprecation of
TLS 1.0 on the way)
- Because an `<input>` element can’t have children, the pseudo-element
hack[2] used to ensure the top edge of the button is clickable doesn’t
work. We’re seeing this bug[3] affect real users in research.
1. We inhereted our buttons from Digital Marketplace, here is me making
that change in their code: 8df7e2e79e (diff-b1420f7b7a25657d849edf90a70ef541)
2. 24e1906c0d (diff-ef0e4eb6f1e90b44b0c3fe39dce274a4R79)
3. https://github.com/alphagov/govuk_elements/issues/545
Spreadsheets start at row 1 (the header row), and the values don’t start
until row 2. The row numbers in our URLs start at 0, which is a concept
that only makes sense to programmers.
It’s more predictable and consistent to make the number in the URL match
the row number displayed on the page when previewing the spreadsheet.
We’ve heard from some users, especially those sending letters, that
they’d like to check that a spreadsheet they’ve uploaded has populated
the template correctly.
My reckon is that seeing just one row of the spreadsheet populate the
template isn’t enough to give people confidence that everything’s
working properly.
This commit adds links to all but the currently-previewed row. Clicking
that link will populate the preview with values from that row. These
pages already exist; they were created in this PR:
https://github.com/alphagov/notifications-admin/pull/1696
We’ve heard from some users, especially those sending letters, that
they’d like to check that a spreadsheet they’ve uploaded has populated
the template correctly.
My reckon is that seeing just one row of the spreadsheet populate the
template isn’t enough to give people confidence that everything’s
working properly. So this commit is the first step towards being able
to preview an arbitrary row of a template, by extending the URL
structure to optionally accept a row number for pages or files (ie PNG)
that preview successfully uploaded spreadsheets.
What this commit doesn’t do is link to these pages; that will come as
part of a subsequent commit.
Easier to test the individual aspects of what’s going on here if they’re
two separate tests, one hitting the initial `POST`, and one hitting the
page that they’re subsequently redirected to.
The `client_request` fixture means the tests take fewer lines of code
and also checks some extra things (eg page titles matching <h1>s).
Changing it in this commit, independently of functional changes to this
test.
Now that the page title for picking a sender/reply to has been improved,
I think these pages are also less clear than they could be.
This commit changes the page titles to (I hope) be clearer about what is
needed from the user on these pages.
Changing the `<h1>` in https://github.com/alphagov/notifications-admin/pull/1638
turned out to be quite confusing. The combination of the word
"recipient" and a selection of email addresses on the page was confusing.
This commit changes the page title to be much more explicit about what
is expected from the page, rather than what is consistent with the text
of the link that the user clicked.
Changing the `<h1>` in https://github.com/alphagov/notifications-admin/pull/1638
turned out to be quite confusing. The combination of the word
"recipient" and a selection of email addresses on the page was confusing.
This commit changes the page title to be much more explicit about what
is expected from the page, rather than what is consistent with the text
of the link that the user clicked.
If you’ve chosen a text message sender then it’s good to see
confirmation of your choice.
This replicates what we do when you choose an email reply-to address.
We have a sort of principle that when clicking a link, the page you land
on should be titled the same as the link you clicked.
This also reduces unnecessary repetition between the page title and the
form label.
Make it clear that:
- In the case of text messages, it’s about who the message comes from
- In the case of emails, it’s about where the user will reply to
If you’ve spelt ‘postcode’ wrong, or missed only ‘address_line_2’ then
it’s pretty noisy to be told that your file needs columns called address
line 1, address line 2, and postcode.
It’s better to be specific about which column you need to fix in order
to get past this error. As a principle, we’ve found it better to tell
get people to fix one error at a time, rather than overwhelm them with a
list of errors to correct – this is why we split the recipient column
errors out separately in the first place.
