werkzeug's internal workings keep files under 500kb in memory, and files
greater than 500kb as a TemporaryFile
(https://github.com/pallets/werkzeug/blob/0.11-maintenance/werkzeug/formparser.py#L38)
when we encounter a CSV or TSV, we call normalise_newlines, which invokes
`.read()`, however when we were passing straight into pyexcel, we called
`file.getvalue()` - this exists on BytesIO (small files) but not on
TemporaryFile objects (large files) - we were seeing 500 errors
In research we’ve seen people mix up the service ID and API key because
they’re both 36 character UUIDs. We can’t get rid of the service ID
because it’s used to look up the API key.
Instead, we should change API key to be one long string, which contains
both the service ID, API key and (optionally) the name of the key. For
example:
```
casework_production-8b3aa916-ec82-434e-b0c5-d5d9b371d6a3-dcdc5083-2fee-4fba-8afd-51f3f4bcb7b0
```
We still need to keep the old, separate, key and service ID for a while
until people have updated their clients. But they’re now both on this
page, rather than on two separate pages, which should make for less
fussing anyway.
This shouldn’t be rolled out until the new clients are available.
- [ ] https://github.com/alphagov/notifications-python-client/pull/36
- [ ] https://github.com/alphagov/notifications-node-client/pull/10
- [ ] https://github.com/alphagov/notifications-ruby-client/pull/15
- [ ] https://github.com/alphagov/notifications-java-client/pull/38
- [ ] PHP????
The GOV.UK content style guide says:
> - 5:30pm (not 1730hrs)
> - midnight (not 00:00)
> - midday (not 12 noon, noon or 12pm)
This commit changes all times to be 12h not 24h, and adds a special case
for when a time is exactly 12:00am or 12:00pm.
This commit makes the CSV download use the same language for failure
reasons as the frontend.
It also adds a test around this stuff, which was patchily tested before.
This is re-fixing a bug which was re-introduced when adding the
`Spreadsheet` class in 1409ca36ca.
It was previously fixed in 19662d8329:
> Fix bug with large file uploads
>
> Depending on the size of the uploaded file, Flask will temporarily store
> it in different ways. This means that it comes back as a `TempFile` if
> the file is roughly <500k and as `BytesIO` if the file is larger.
>
> `TempFile` supports the `.getvalue()` method, but `BytesIO` does not.
> Both support the `.read()` method, so this commit changes to use that
> instead.
Our templates are a littered with `request.args.get('help', '0')`.
This commit refactors these into a single helper method, which can be
used by the view functions, then passed to the template.
This makes the templates cleaner, and should make it easier to refactor
`help` out of the query parameters entirely in the future.
parts of the initial setup/login stages were throwing 500s if user
not already in process (ie: user directly navigated to url):
* /resend-email-verification
* /text-not-received
* /send-new-code
* verify
This commit rearranges the CSV errors (again) to make them geared towards
teaching you how to match placeholders to the column headers.
So the order of errors now is:
1. No phone number/email column
2. Column headers don’t match placeholders
3. Missing or bad data
4. Trial mode
5. Daily limit reached
This depends on:
- [x] https://github.com/alphagov/notifications-utils/pull/39 for 1.
We require users to export their spreadsheets as CSV files before
uploading them. But this seems like the sort of thing a computer should
be able to do.
So this commit adds a wrapper class which:
- takes a the uploaded file
- returns it in a normalised format, or reads it using pyexcel[1]
- gives the data back in CSV format
This allows us to accept `.csv`, `.xlsx`, `.xls` (97 and 95), `.ods`,
`.xlsm` and `.tsv` files. We can upload the resultant CSV just like
normal, and process it for errors as before.
Testing
---
To test this I’ve added a selection of common spreadsheet files as test
data. They all contain the same data, so the tests look to see that the
resultant CSV output is the same for each.
UI changes
---
This commit doesn’t change the UI, apart from to give a different error
message if a user uploads a file type that we still don’t understand.
I intend to do this as a separate pull request, in order to fulfil
https://www.pivotaltracker.com/story/show/119371637
Each service on the list is linked to the dashboard page of the service.
The platform admin user can see/edit templates, see/invite users, see/edit service settings.
The platform admin user can not send messages, see/edit api keys and developer docs.
Makes uses of the additions to utils in https://github.com/alphagov/notifications-utils/pull/9
This commit strips out a lot of the complex stuff that the views and templates
in this app were doing. There is now a cleaner separation of concerns:
- utils returns the number and type of errors in the csv
- `get_errors_for_csv` helper in this app maps the number and type of errors
onto human-friendly error messages
- the view and template just doing the glueing-together of all the pieces
This is (hopefully) easier to understand, definitely makes the component
parts easier to test in isolation, and makes it easier to give more specific
error messages.
CSV files currently have ‘to’ as the recipient column. This is changing in
https://github.com/alphagov/notifications-api/pull/109
The admin app also has to validate that the CSV files have the right columns,
because the API expects any CSV that it’s given to have been checked (also we
want things to actually work).
This commit is the minimum code change needed. In the future it should reuse
the same code as the API for processing CSV files. This will need more thinking.
This commit parameterises all methods in the send view so that they can send
either emails or SMS messages.
It works out what kind of message it is sending from the `template_type`
property of the template object.
This means that the `Template` util class needs to know about these properties,
which means that this commit depends on:
https://github.com/alphagov/notifications-utils/pull/2
This commit does _not_ add tests for sending emails. The existing tests for
sending SMS still pass, but actually sending emails is outside the scope of
this story.
This commit extends the existing function to validate each row’s phone number
to also validate that all the required data is present.
It does this using the checking that the `Template` class can do when given
a template and a `dict` of values.
