This changeset pulls in all of the notification_utils code directly into the admin and removes it as an external dependency. We are doing this to cut down on operational maintenance of the project and will begin removing parts of it no longer needed for the admin.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
- Deleted /stylesheets folder
- Removed sass build from gulpfile
- Changed gov links to usa links
- Changed other govuk styles, like breadcrumbs
- Changed name of uk_components file to us_components
- Fixed a few tests that broke on account of the changes
To render text in an SVG consistently the system rendering the SVG must
have the fonts specified by the SVG installed.
If the fonts are not installed then the renderer will fall back to a
system font and the text will look different. This is especially bad
news for branding where the right font is an integral part of any brand.
To fix this, the text should instead be converted to `<path>` elements.
This process is sometimes called ‘outlining’.
A few of our logos had this problem, and I’ve fixed most of them by
hand. Adding this validation will stop the problem, coming up again.
WTForms versions less than 3.0.0 have a security vulnerability where
arbitrary HTML can be inserted into the label of a form, allowing the
possibility of a cross-site scripting attack.
I don’t know if there’s anywhere we put user-generated content into form
labels but it’s possible we are vulnerable somewhere.
This require moving some imports because as of
https://github.com/wtforms/wtforms/pull/614/files
there is no longer a separate module for HTML 5 fields, they are now
considered core fields.
As of https://github.com/wtforms/wtforms/issues/445/files custom
implementations of `pre_validate` or `post_validate` must raise
`ValidationError` to trigger a validation message, where we were raising
`ValueError` this was no longer being caught.
As of https://github.com/wtforms/wtforms/pull/355/files `StringField`
returns `None` for empty data, not `''` but our `validate_email_address`
function only accepts strings.
This is so we can allow the sender name 'UC' for DWP.
Note, this is specifically only straight single quotes and not curly
quotes or double quotes. Curly quotes are not supported in the GSM
character set (https://en.wikipedia.org/wiki/GSM_03.38). There is
currently no defined user ask to support double quotes in sms sender
names.
I have tested this by sending a message through both Firetext and MMG to
make sure they both support the single quote character in SMS sender
names.
DWP also have had no particular issues using the SMS sender name with
their existing system in the past either.
`app/utils.py` is a bit of a dumping ground for things we don’t have a
better place for.
We now have a place and structure for storing ‘model’ code (‘model’ in
the model, view, controller (MVC) sense of the word).
This commit moves the spreadsheet model to that place.
Depends on:
- [ ] https://github.com/alphagov/notifications-utils/pull/826/files
Adds error messages for when the content of a broadcast template is too
long.
The error message is explicit when this is cause by non-GSM characters.
We may not want to expose this complexity to our users, but it’s useful
for now while we’re testing things out.
‘Commonly used passwords’ is more specific, and avoids the terminology
‘blacklist’ which the National Cyber Security Centre explain to be
problematic:
> It's fairly common to say whitelisting and blacklisting to describe desirable and undesirable things in cyber security. For instance, when talking about which applications you will allow or deny on your corporate network; or deciding which bad passwords you want your users not to be able to use.
>
> However, there's an issue with the terminology. It only makes sense if you equate white with 'good, permitted, safe' and black with 'bad, dangerous, forbidden'. There are some obvious problems with this. So in the name of helping to stamp out racism in cyber security, we will
> avoid this casually pejorative wording on our website in the future. No, it's not the biggest issue in the world - but to borrow a slogan from elsewhere: every little helps.
– https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
This involves three changes which broke our code.
To validate email addresses, the optional dependency `email-validator`
must be installed<sup>1</sup>. But since we don’t use WTForms’ email
validation, we shouldn’t need to subclass it – it can just be its own
self contained thing. Then we don’t need to add the extra dependency.
When rendering textareas, and extra `\r\n` is inserted at the beginning
<sup>2</sup>. Browsers will strip this when displaying the textbox and
submitting the form, but some of our tests need updating to account for
this.
The error message for when you don’t choose an option from some radio
buttons has now changed. Rather than just accepting WTForms’ new
message, this commit makes the error messages like the examples from
the Design System<sup>3</sup>. By default it will say ‘Select an
option’, but by passing in an extra parameter (`thing`) it can be
customised to be more specific, for example ‘Select a type of
organisation’.
***
1. https://github.com/wtforms/wtforms/pull/429
2. https://github.com/wtforms/wtforms/issues/238
3. https://design-system.service.gov.uk/components/radios/#error-messages
If someone enters an email address from a domain we don’t recognise we
direct them straight to our support channel. This is causing increased
contact from suppliers and members of the public.
Now that we have a page which explains who can use Notify, let’s direct
people there first. Then if they really do need to contact support
(because we don’t recognise their organisation) then they can do so from
that page.
service contact blocks contain new lines - and jinja2 normally ignores
newlines (as in it keeps them as new lines) - but we need to turn them
into `<br>` tags so that we can show the formatting that the user has
added. We were previously just doing `{{ block | nl2br | safe }}`. nl2br
turns the new lines into `<br>` tags, and then `safe` tells jinja that
it doesn't need to escape the html.
this causes issues if the user adds `<script>alert(1)</script>` to their
contact block (or some other evil xss hack), where that will get let
through due to the safe flag
To solve this, use `Markup(html='escape')` to sanitise any html, and
then convert new lines to <br>.
bump utils
another xss
- This brings in the latest version of notifications-utils which
allows Welsh characters in SMS templates.
- Updated the pricing page to show the new prices for SMS with certain
Welsh characters
We’re deprecating storing the domain as text on a branding in favour of
a database relationship between branding and organisation.
We need to do this now in order to remove the validation on these fields
(which depends on the data in `domains.yml`)
Otherwise we can end up collecting invalid email addresses…
This required some refactoring to allow our email fields to be optional
(but not by default).
At the moment we transform what the user gives us, so if someone enters
`digital.cabinet-office.gov.uk` it will automagically be saved as
`cabinet-office.gov.uk`. This happens without the user knowing, and
might have unintended consequences.
So let’s tell them what the problem is, and let them decide what to do
about it (which might be accepting the canonical domain, or adding a
new organisation to domains.yml first).
We should make sure we’re not putting typos in the branding list. We can
validate what gets entered here against our known list of public-sector
domains.
Updated notifications-utils. This brings in
- the renamed character sanitization classes
- the change to allow unicode in letter addresses (this lets us delete
a test that is no longer relevant)
Also replaced non-ascii characters in headers. This fixes a bug where
non-ascii characters in a CSV filename were causing errors since the
filename is also used in the header.
Having SMS senders that start with 00 can cause issues with Firetext due
to Firetext's validation rules, so we shouldn't allow SMS senders to start
with 00.
Firetext treats a double 00 at the start of the senderID as an international
prefix, so removes them. A sender of 00447876574016 would become 447876574016.
Under Firetext's validation rules, an SMS sender of five 0s (00000) would
become 4400. This is because the first 00 are removed (as the international
prefix). The third 0 is seen as the start of a phone number, and becomes 44,
leaving the final 00 = 4400.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
previously we were just using the wtforms builtin email validator,
which is much more relaxed than our own one. It'd catch bad emails when
POSTing to the API, resulting in an ugly error message. It's easy work
to make sure we validate email addresses as soon as they're entered.
We have a team who want their (short) web address as the text message
sender. This commit updates the validation of text message senders to
allow `.` as a valid character, which is currently blocking them from
doing this.
We can be fairly confident this works because:
- the team are sending large volumes of messages already with their
existing provider
- we’ve tested it with all combinations of
- both our text message providers
- an Android phone and n iPhone
Using a separate validator class to check for appropriate characters in
a text message sender means that we’re not doing this validation in a
different way from the other checks (length and required). So the code
is cleaner.
We call the yellow things ‘double brackets’ on the frontend, not fields
or placeholders. This error message was a bit out of date.
Also refactored it to use the `Field` class; this code was probably
written before `Field` was factored out of `Template`.
Use `it`/`they` depending on how many different characters you've used
Also don't wrap the message with quotes, as it looks confusing and
potentialy implies that you can't use apostrophes
The kind of communications we’re getting at the moment can broadly be
broken down into:
- problems
- questions and feedback
We will need to triage problems differently, because they could
potentially be urgent/severe/emergency/P1/whatever language we use.
Questions or feedback will never be P1.
Two reasons for making the user categorise their tickets themselves:
- Outside of hours we can’t get someone out of bed in order to decide if
a ticket is a problem or just feedback
- We can tailor the subsequent pages to whether it’s a problem or
feedback (eg showing a link to the status page if the user is having
a problem)
This commit let’s users make the choice with a pair of radio buttons.
It also cleans up a bunch of the tests and parameterizes them so we’re
testing the flow for both ticket types.
