See c31264d4c for why ‘whitelist’ should be avoided. The use of
whitelist here was not referring to the user-maintained list, but to
mean ‘not a government’ email address. This commit renames these tests
to make that difference clear.
This involves three changes which broke our code.
To validate email addresses, the optional dependency `email-validator`
must be installed<sup>1</sup>. But since we don’t use WTForms’ email
validation, we shouldn’t need to subclass it – it can just be its own
self contained thing. Then we don’t need to add the extra dependency.
When rendering textareas, and extra `\r\n` is inserted at the beginning
<sup>2</sup>. Browsers will strip this when displaying the textbox and
submitting the form, but some of our tests need updating to account for
this.
The error message for when you don’t choose an option from some radio
buttons has now changed. Rather than just accepting WTForms’ new
message, this commit makes the error messages like the examples from
the Design System<sup>3</sup>. By default it will say ‘Select an
option’, but by passing in an extra parameter (`thing`) it can be
customised to be more specific, for example ‘Select a type of
organisation’.
***
1. https://github.com/wtforms/wtforms/pull/429
2. https://github.com/wtforms/wtforms/issues/238
3. https://design-system.service.gov.uk/components/radios/#error-messages
A lot of pages in the admin app are now generated entirely from Redis,
without touching the API.
The one remaining API call that a lot of pages make, when the user is
platform admin or a member of an organisation, is to get the name of
the current service’s organisation.
This commit adds some code to start caching that as well, which should
speed up page load times for when we’re clicking around the admin app
(it’s typically 100ms just to get the organisation, and more than that
when the API is under load).
This means changing the service model to get the organisation from the
API by ID, not by service ID. Otherwise it would be very hard to clear
the cache if the name of the organisation ever changed.
We can’t cache the whole organisation because it has a
`count_of_live_services` field which can change at any time, without an
update being made.
Although their allowances are the same as what we call `nhs_local` it
makes more sense to store them separately because:
- we already present them as two separate choices to the user
- we may want to handle them differently in the future, eg in terms of
what branding choices are available to them
Once the API is updated we can start passing in this new value from
the admin app.
The data flow of other bits of our application looks like this:
```
API (returns JSON)
⬇
API client (returns a built in type, usually `dict`)
⬇
Model (returns an instance, eg of type `Service`)
⬇
View (returns HTML)
```
The user API client was architected weirdly, in that it returned a model
directly, like this:
```
API (returns JSON)
⬇
API client (returns a model, of type `User`, `InvitedUser`, etc)
⬇
View (returns HTML)
```
This mixing of different layers of the application is bad because it
makes it hard to write model code that doesn’t have circular
dependencies. As our application gets more complicated we will be
relying more on models to manage this complexity, so we should make it
easy, not hard to write them.
It also means that most of our mocking was of the User model, not just
the underlying JSON. So it would have been easy to introduce subtle bugs
to the user model, because it wasn’t being comprehensively tested. A lot
of the changed lines of code in this commit mean changing the tests to
mock only the JSON, which means that the model layer gets implicitly
tested.
For those reasons this commit changes the user API client to return
JSON, not an instance of `User` or other models.
At the moment we have to update a YAML file and deploy the change to get
a new domain whitelisted.
We already have a thing for adding new domains – the organisation stuff.
This commit extends the validation to look in the `domains` table on the
API if it can’t find anything in the YAML whitelist.
This has the advantage of:
- not having to deploy code to whitelist a new domain
- forcing us to create new organisations as they come along, so that
users’ services automatically get allocated to the organisation once
their domain is whitelisted
If we can’t guess a user’s organisation then they have to choose what
type of organisation they work for when they add a new service. This
commit adds a test to make sure they can’t just click the green button
and proceed without picking one.
We get people signing up for Notify who work for the NHS, but whose
organisation we don’t know about. For example
`name@gloshospitals.nhs.uk` will be someone working for Gloucestershire
Hospitals NHS Foundation Trust, which is not an organisation we have in
the database.
Currently we rely on knowing the specific organisation (NHS as a whole
isn’t an organisation) in order to set the organisation type for any
services they create. This commit adds a special case for anyone with an
NHS email address to set the organisation type to be NHS, even when we
don’t know which specific part of the NHS they work for.
This is the same thing we do on the API side for NHS email and letter
branding:
a4ae5a0a90/app/dao/services_dao.py (L310-L313)
An ‘unknown’ organisation can either be:
- one where we know it exists but don’t know much about it (in which
case the API returns some JSON with the info we do know)
- one we’ve never come across (in which case the API will return `None`)
This commit fixes a bug where we were trying to access the organisation
type in the latter case.
Every time someone adds a new service we ask them what kind of
organisation they work for.
We can look this up based on the user’s email address now. So we should
only ask the question if:
- we don’t know about the organisation
- or we haven’t set what type of organisation it is (this shouldn’t be
possible on productions because we’ve populated the column for all
existing organisations and it’s impossible to add a new one without
setting it
It:
- saves repetetive boilerplate code
- does some extra checks (eg checking for a `200` response)
- makes the codebase less confusing to consistently do the same thing in
the same way
Anyone choosing ‘NHS’ for their organisation type gets should get the
NHS branding. We don’t want to hard-code an ID for NHS branding anywhere
because it won’t be consistent between environments.
So instead we can say that anyone who chooses ‘NHS’ as their
organisation type should get whatever branding has `nhs.uk` as its
domain.
This allows us to easily manage the branding the same way we do other
brands, but gives us the efficiency of having it auto applied.
When a user creates a service we can take a pretty good guess at what
organisation they’re from.
For many organisations, especially local councils, GOV.UK branding is
not appropriate for their service. But right now every service:
- gets created with GOV.UK branding
- has to ask us to change it, even if they’ve already done so for other
services they run
This commit starts using the `domain` field on the email branding table
to lookup what email branding to assign to a service automatically,
where we’re sure there’s a sensible default.
Since we send all one off messages as priority now[1], we don’t need to
explicitly mark this template as being priority.
This stops the (potential) problem of people skipping the tour, still
having this template and then modifying it to send other messages,
potentially in high volumes from CSV files or the API. I don’t think
this is a real problem now, but worth cleaning this up.
Currently:
- 827 priority templates in the database
- 195 of which are not deleted
- 18 of which are not called ‘Example text message template’
- 3 of which look like genuine use cases, not from services that we run
[1]: https://github.com/alphagov/notifications-api/pull/1722
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
Different parts of government get billed slightly differently, and
there’s differences in how much money we’re allowed to give them.
Think these numbers are right, but should be double checked.
So that we can default services to their appropriate text allowance, we
need to find out what sector they're in. So let's start collecting that
from teams as they create new services.
I think Central/Local/NHS are the right options, but these can be easily
changed if not.
We’ve made a few changes to the tour recently, without changing the
help text on the left hand side of the screen. So the stuff you see on
the right side of the screen doesn’t quite sync up any more.
This commit adds an extra, introductory page that just shows the
template and a next button, which better matches the ‘every message
starts with a template’ help text.
The tour teaches you how Notify works by letting you do a thing and then
showing you the effect of the thing you’ve just done – a text message on
your phone.
This is not as effective if you don’t get the text message quickly. It
breaks the association you make between what you’ve done on the computer
and what’s happening on your phone.
Slow text message delivery can happen if you’re doing a big job. We can
get around this by making your text message use the priority queue.
This was observed in the pilot research session yesterday.
We have a bunch of different styles of handling when function
definitions span multiple lines, which they almost always do with tests.
Here’s why an argument per line, single indent is best:
- cleaner diffs when you change the name of a method (one line change
instead of multiple lines)
- works better on narrow screens, eg Github’s diff view, or with two
terminals side by side on a laptop screen
- works with any editor’s indenting shortcuts, no need for an IDE
Also, trailing comma in the list of arguments is good because adding a
new argument to a method becomes a one line, not two line diff.
The test for non-gov.uk domains adding services is still relevant, but
probably makes more sense in `test_add_services.py`.
The others are no longer relevant now the ‘All services’ page has gone.
We reckon that:
- a ‘blank slate’ templates page is a better start
- the example template hurt more than it helped when it comes to
understanding placeholders
_The code for this is quite hacky and light on tests. But I’d really like to get
it in the app for the research tomorrow to see how well the feature works._
This commit changes the tour from being a set of static screens to some help
which guides you through the process of sending your first test message.
The theory behind this is that what users are really struggling with is the
concept of a variable, rather than the relationship between the placeholders and
the column headers. And like learning to program, the best way to learn is by
taking an example and modifying it to your own needs.
This means that when someone adds their first service we set them up an
example email template and an example text message template. Then there is a
guided, three step process where _all_ the user can do is send a test message to
themselves.
Once the message is sent, the user still has the example templates which they
can edit, rather than having to remember what they’re supposed to be doing.
When you add a new service, it’s probably the one you want to do stuff
with.
When you get invited, the service you’ve been invited to is probably the
one you want to use.
This commit adds the ID of the new service or service you’ve been
invited to to the session.
