We have a `client_request` fixture which does a bunch of useful stuff
like:
- checking the status code of the response
- returning a `BeautifulSoup` object
Lots of our tests still use an older fixture called `client`. This is
not as good because it:
- returns a raw `Response` object
- doesn’t do the additional checks
- means our tests contain a lot of repetetive boilerplate like `page = BeautifulSoup(response.data.decode('utf-8'), 'html.parser')`
This commit converts all the tests which had a `client.login(…)`
statement to use `client_request` (which is already logged in by
default).
Subsequent commits will remove uses of `client` in other tests, but
doing it this way means the work can be broken up into more manageable
chunks.
This is the newest version.
Pyup is complaining about vulnerabilities in version 1.0.1, specifically
> Werkzeug version 2.0.2 improves the security of the debugger cookies.
> "SameSite" attribute is set to "Strict" instead of "None", and the
> secure flag is added when on HTTPS.
Previously we were using whatever version of Werkzeug that Flask
specified this pins it to get rid of the vulnerability without having to
upgrade everything at once.
This requires a few changes to tests which were relying on importing
`session` and `current_user` from Flask. Previously it seemed that
importing these in the tests referred to the same object that was being
used in the app. This appears to no longer be the case. This commit
works around that by:
- using a context manager to get the contents of the session, like we
already do in most tests
- asserting that the mock which logs the user in is being called with
the right values, rather than looking at the state of the
`current_user` object (which was probably giving false certainty
anyway)
This naming was introduced in 2016 without explanation [1]. I find it
confusing because:
- It's reminiscent of "_app", which is a Python convention indicating
the variable is internal, so maybe avoid using it.
- It suggests there's some other "app" fixture I should be using (there
isn't, though).
The Python style guide describes using an underscore suffix to avoid
clashes with inbuilt names [1], which is sort of applicable if we need
to import the "app" module [2]. However, we can also avoid clashes by
choosing a different name, without the strange underscore.
[1]: 3b1d521c10
[2]: 78824f54fd/tests/app/main/views/test_forgot_password.py (L5)
The API has a method to handle setting the default SMS free allowance. This will save a call to the API and remove some code duplication between the two apps.
Needs to be merged after https://github.com/alphagov/notifications-api/pull/3197
🚨 Do not merge until after 1 April 2020 🚨
Once this date has past we no longer need to give any services the
previous allowances, so we can remove them from the codebase to avoid
confusion.
It’s possible we change the allowance structure again, but it might
change in a way that this config-based logic doesn’t account for (what
if we did a per-organisation allowance for example). Having both years’
allowances in the config was a quick fix, not a foundation to build on.
We’re going to have different allowances next financial year. This means
that when someone adds a service, we’ll need to check which year it is,
so we can give them the right allowance.
This commit changes the config structure so that the current allowances
are explicitly assigned to the 2020/21 financial year.
It freezes the tests to the 2020/21 financial year, so they won’t start
failing automatically when next financial year comes around.
There was a recent error in the logs because a service tried to change
its name to one exceeding 255 characters (which is a limit on the
database field). We can easily catch these errors on the form, so that
the user doesn't see an error page.
The `post` method of the `client_request` fixture has an argument called
`_data`. There were a few places where we had used an argument of `data`
instead by mistake.
Changes the OrganisationTypeField class used by
OrganisationOrganisationTypeForm.organisation_type
OrganisationTypeField is also used by the forms in
/add-service:
- CreateServiceForm
- CreateNhsServiceForm
Because of that, this commit also includes changes
to the template for that route.
Note: this also moves where OrganisationTypeField
appears in app/main/forms.py so it can use
GovukRadiosField.
See c31264d4c for why ‘whitelist’ should be avoided. The use of
whitelist here was not referring to the user-maintained list, but to
mean ‘not a government’ email address. This commit renames these tests
to make that difference clear.
This involves three changes which broke our code.
To validate email addresses, the optional dependency `email-validator`
must be installed<sup>1</sup>. But since we don’t use WTForms’ email
validation, we shouldn’t need to subclass it – it can just be its own
self contained thing. Then we don’t need to add the extra dependency.
When rendering textareas, and extra `\r\n` is inserted at the beginning
<sup>2</sup>. Browsers will strip this when displaying the textbox and
submitting the form, but some of our tests need updating to account for
this.
The error message for when you don’t choose an option from some radio
buttons has now changed. Rather than just accepting WTForms’ new
message, this commit makes the error messages like the examples from
the Design System<sup>3</sup>. By default it will say ‘Select an
option’, but by passing in an extra parameter (`thing`) it can be
customised to be more specific, for example ‘Select a type of
organisation’.
***
1. https://github.com/wtforms/wtforms/pull/429
2. https://github.com/wtforms/wtforms/issues/238
3. https://design-system.service.gov.uk/components/radios/#error-messages
A lot of pages in the admin app are now generated entirely from Redis,
without touching the API.
The one remaining API call that a lot of pages make, when the user is
platform admin or a member of an organisation, is to get the name of
the current service’s organisation.
This commit adds some code to start caching that as well, which should
speed up page load times for when we’re clicking around the admin app
(it’s typically 100ms just to get the organisation, and more than that
when the API is under load).
This means changing the service model to get the organisation from the
API by ID, not by service ID. Otherwise it would be very hard to clear
the cache if the name of the organisation ever changed.
We can’t cache the whole organisation because it has a
`count_of_live_services` field which can change at any time, without an
update being made.
Although their allowances are the same as what we call `nhs_local` it
makes more sense to store them separately because:
- we already present them as two separate choices to the user
- we may want to handle them differently in the future, eg in terms of
what branding choices are available to them
Once the API is updated we can start passing in this new value from
the admin app.
The data flow of other bits of our application looks like this:
```
API (returns JSON)
⬇
API client (returns a built in type, usually `dict`)
⬇
Model (returns an instance, eg of type `Service`)
⬇
View (returns HTML)
```
The user API client was architected weirdly, in that it returned a model
directly, like this:
```
API (returns JSON)
⬇
API client (returns a model, of type `User`, `InvitedUser`, etc)
⬇
View (returns HTML)
```
This mixing of different layers of the application is bad because it
makes it hard to write model code that doesn’t have circular
dependencies. As our application gets more complicated we will be
relying more on models to manage this complexity, so we should make it
easy, not hard to write them.
It also means that most of our mocking was of the User model, not just
the underlying JSON. So it would have been easy to introduce subtle bugs
to the user model, because it wasn’t being comprehensively tested. A lot
of the changed lines of code in this commit mean changing the tests to
mock only the JSON, which means that the model layer gets implicitly
tested.
For those reasons this commit changes the user API client to return
JSON, not an instance of `User` or other models.
At the moment we have to update a YAML file and deploy the change to get
a new domain whitelisted.
We already have a thing for adding new domains – the organisation stuff.
This commit extends the validation to look in the `domains` table on the
API if it can’t find anything in the YAML whitelist.
This has the advantage of:
- not having to deploy code to whitelist a new domain
- forcing us to create new organisations as they come along, so that
users’ services automatically get allocated to the organisation once
their domain is whitelisted
If we can’t guess a user’s organisation then they have to choose what
type of organisation they work for when they add a new service. This
commit adds a test to make sure they can’t just click the green button
and proceed without picking one.
We get people signing up for Notify who work for the NHS, but whose
organisation we don’t know about. For example
`name@gloshospitals.nhs.uk` will be someone working for Gloucestershire
Hospitals NHS Foundation Trust, which is not an organisation we have in
the database.
Currently we rely on knowing the specific organisation (NHS as a whole
isn’t an organisation) in order to set the organisation type for any
services they create. This commit adds a special case for anyone with an
NHS email address to set the organisation type to be NHS, even when we
don’t know which specific part of the NHS they work for.
This is the same thing we do on the API side for NHS email and letter
branding:
a4ae5a0a90/app/dao/services_dao.py (L310-L313)
An ‘unknown’ organisation can either be:
- one where we know it exists but don’t know much about it (in which
case the API returns some JSON with the info we do know)
- one we’ve never come across (in which case the API will return `None`)
This commit fixes a bug where we were trying to access the organisation
type in the latter case.
Every time someone adds a new service we ask them what kind of
organisation they work for.
We can look this up based on the user’s email address now. So we should
only ask the question if:
- we don’t know about the organisation
- or we haven’t set what type of organisation it is (this shouldn’t be
possible on productions because we’ve populated the column for all
existing organisations and it’s impossible to add a new one without
setting it
It:
- saves repetetive boilerplate code
- does some extra checks (eg checking for a `200` response)
- makes the codebase less confusing to consistently do the same thing in
the same way
Anyone choosing ‘NHS’ for their organisation type gets should get the
NHS branding. We don’t want to hard-code an ID for NHS branding anywhere
because it won’t be consistent between environments.
So instead we can say that anyone who chooses ‘NHS’ as their
organisation type should get whatever branding has `nhs.uk` as its
domain.
This allows us to easily manage the branding the same way we do other
brands, but gives us the efficiency of having it auto applied.
When a user creates a service we can take a pretty good guess at what
organisation they’re from.
For many organisations, especially local councils, GOV.UK branding is
not appropriate for their service. But right now every service:
- gets created with GOV.UK branding
- has to ask us to change it, even if they’ve already done so for other
services they run
This commit starts using the `domain` field on the email branding table
to lookup what email branding to assign to a service automatically,
where we’re sure there’s a sensible default.
Since we send all one off messages as priority now[1], we don’t need to
explicitly mark this template as being priority.
This stops the (potential) problem of people skipping the tour, still
having this template and then modifying it to send other messages,
potentially in high volumes from CSV files or the API. I don’t think
this is a real problem now, but worth cleaning this up.
Currently:
- 827 priority templates in the database
- 195 of which are not deleted
- 18 of which are not called ‘Example text message template’
- 3 of which look like genuine use cases, not from services that we run
[1]: https://github.com/alphagov/notifications-api/pull/1722
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
Different parts of government get billed slightly differently, and
there’s differences in how much money we’re allowed to give them.
Think these numbers are right, but should be double checked.
So that we can default services to their appropriate text allowance, we
need to find out what sector they're in. So let's start collecting that
from teams as they create new services.
I think Central/Local/NHS are the right options, but these can be easily
changed if not.
