service contact blocks contain new lines - and jinja2 normally ignores
newlines (as in it keeps them as new lines) - but we need to turn them
into `<br>` tags so that we can show the formatting that the user has
added. We were previously just doing `{{ block | nl2br | safe }}`. nl2br
turns the new lines into `<br>` tags, and then `safe` tells jinja that
it doesn't need to escape the html.
this causes issues if the user adds `<script>alert(1)</script>` to their
contact block (or some other evil xss hack), where that will get let
through due to the safe flag
To solve this, use `Markup(html='escape')` to sanitise any html, and
then convert new lines to <br>.
bump utils
another xss
- This brings in the latest version of notifications-utils which
allows Welsh characters in SMS templates.
- Updated the pricing page to show the new prices for SMS with certain
Welsh characters
We’re deprecating storing the domain as text on a branding in favour of
a database relationship between branding and organisation.
We need to do this now in order to remove the validation on these fields
(which depends on the data in `domains.yml`)
Otherwise we can end up collecting invalid email addresses…
This required some refactoring to allow our email fields to be optional
(but not by default).
At the moment we transform what the user gives us, so if someone enters
`digital.cabinet-office.gov.uk` it will automagically be saved as
`cabinet-office.gov.uk`. This happens without the user knowing, and
might have unintended consequences.
So let’s tell them what the problem is, and let them decide what to do
about it (which might be accepting the canonical domain, or adding a
new organisation to domains.yml first).
We should make sure we’re not putting typos in the branding list. We can
validate what gets entered here against our known list of public-sector
domains.
Updated notifications-utils. This brings in
- the renamed character sanitization classes
- the change to allow unicode in letter addresses (this lets us delete
a test that is no longer relevant)
Also replaced non-ascii characters in headers. This fixes a bug where
non-ascii characters in a CSV filename were causing errors since the
filename is also used in the header.
Having SMS senders that start with 00 can cause issues with Firetext due
to Firetext's validation rules, so we shouldn't allow SMS senders to start
with 00.
Firetext treats a double 00 at the start of the senderID as an international
prefix, so removes them. A sender of 00447876574016 would become 447876574016.
Under Firetext's validation rules, an SMS sender of five 0s (00000) would
become 4400. This is because the first 00 are removed (as the international
prefix). The third 0 is seen as the start of a phone number, and becomes 44,
leaving the final 00 = 4400.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
previously we were just using the wtforms builtin email validator,
which is much more relaxed than our own one. It'd catch bad emails when
POSTing to the API, resulting in an ugly error message. It's easy work
to make sure we validate email addresses as soon as they're entered.
We have a team who want their (short) web address as the text message
sender. This commit updates the validation of text message senders to
allow `.` as a valid character, which is currently blocking them from
doing this.
We can be fairly confident this works because:
- the team are sending large volumes of messages already with their
existing provider
- we’ve tested it with all combinations of
- both our text message providers
- an Android phone and n iPhone
Using a separate validator class to check for appropriate characters in
a text message sender means that we’re not doing this validation in a
different way from the other checks (length and required). So the code
is cleaner.
We call the yellow things ‘double brackets’ on the frontend, not fields
or placeholders. This error message was a bit out of date.
Also refactored it to use the `Field` class; this code was probably
written before `Field` was factored out of `Template`.
Use `it`/`they` depending on how many different characters you've used
Also don't wrap the message with quotes, as it looks confusing and
potentialy implies that you can't use apostrophes
The kind of communications we’re getting at the moment can broadly be
broken down into:
- problems
- questions and feedback
We will need to triage problems differently, because they could
potentially be urgent/severe/emergency/P1/whatever language we use.
Questions or feedback will never be P1.
Two reasons for making the user categorise their tickets themselves:
- Outside of hours we can’t get someone out of bed in order to decide if
a ticket is a problem or just feedback
- We can tailor the subsequent pages to whether it’s a problem or
feedback (eg showing a link to the status page if the user is having
a problem)
This commit let’s users make the choice with a pair of radio buttons.
It also cleans up a bunch of the tests and parameterizes them so we’re
testing the flow for both ticket types.
If a user chooses a very common password then an attacker could guess it
in relatively few attempts, circumventing the lockout.
CESG recommend blacklisting the most common passwords:
> …enforcing the requirement for complex character sets in passwords is
> not recommended. Instead, concentrate efforts on technical controls,
> especially:
>
> - defending against automated guessing attacks by either using account
> lockout, throttling, or protective monitoring
> - blacklisting the most common password choices
How I made this list:
- went to the OWASP repository of security lists:
https://github.com/danielmiessler/SecLists
- downloaded `10k_most_common.txt`, `twitter-banned.txt` and
`500-worst-passwords.txt`
- filtered out any under 8 characters:
```
sed -r '/^.{,7}$/d' passwords-twitter.txt > passwords-combined.txt
sed -r '/^.{,7}$/d' passwords-500.txt >> passwords-combined.txt
sed -r '/^.{,7}$/d' passwords.txt >> passwords-combined.txt
```
- filtered out any duplicates:
```
cat passwords-combined.txt | awk '!x[$0]++' > passwords-combined-deduped.txt
```
We require users to export their spreadsheets as CSV files before
uploading them. But this seems like the sort of thing a computer should
be able to do.
So this commit adds a wrapper class which:
- takes a the uploaded file
- returns it in a normalised format, or reads it using pyexcel[1]
- gives the data back in CSV format
This allows us to accept `.csv`, `.xlsx`, `.xls` (97 and 95), `.ods`,
`.xlsm` and `.tsv` files. We can upload the resultant CSV just like
normal, and process it for errors as before.
Testing
---
To test this I’ve added a selection of common spreadsheet files as test
data. They all contain the same data, so the tests look to see that the
resultant CSV output is the same for each.
UI changes
---
This commit doesn’t change the UI, apart from to give a different error
message if a user uploads a file type that we still don’t understand.
I intend to do this as a separate pull request, in order to fulfil
https://www.pivotaltracker.com/story/show/119371637
> If a user tries to save a template containing something like
> ((name,date)) we should give a validation error.
This is because it causes havoc with the column headers in CSV files.
https://www.pivotaltracker.com/story/show/117043389
At the moment the file contents are not persisted by checked in
memory.
The first and last three records are show if all are valid.
If there are invalid rows, they are reported and the user is
prompted to go back and sort out upload file.
The storing of upload result (i.e. validation of file) in session
will be removed in next story which is about persisting of file
for later processing.
