We were using user fixtures in a lot of parameterized tests, but this is
no longer allowed in Pytest 5. To avoid having to split up the parametrized
tests (which would make the test files a lot longer and slightly more
difficult to read) this commit creates functions which return various types
of user json so that we can use these as the test parameters instead.
Before, the delete decorator would delete the keys from Redis and then
we made the request to api to change the data. However, it is possible
that the cache could get re-populated in between these two things
happening, and so would cache outdated data.
This changes the order to send the api request first. We then always
delete the specified keys from Redis. Changing the order of the code in
the decorator changes the order in which the cache keys get deleted, so
the tests have been updated.
Users can only be archived by Platform Admin from the user page
(/users/<user_id>). This removes them from all services and orgs and
updates their details.
The data flow of other bits of our application looks like this:
```
API (returns JSON)
⬇
API client (returns a built in type, usually `dict`)
⬇
Model (returns an instance, eg of type `Service`)
⬇
View (returns HTML)
```
The user API client was architected weirdly, in that it returned a model
directly, like this:
```
API (returns JSON)
⬇
API client (returns a model, of type `User`, `InvitedUser`, etc)
⬇
View (returns HTML)
```
This mixing of different layers of the application is bad because it
makes it hard to write model code that doesn’t have circular
dependencies. As our application gets more complicated we will be
relying more on models to manage this complexity, so we should make it
easy, not hard to write them.
It also means that most of our mocking was of the User model, not just
the underlying JSON. So it would have been easy to introduce subtle bugs
to the user model, because it wasn’t being comprehensively tested. A lot
of the changed lines of code in this commit mean changing the tests to
mock only the JSON, which means that the model layer gets implicitly
tested.
For those reasons this commit changes the user API client to return
JSON, not an instance of `User` or other models.
The api endpoint to get all template folders also returns the users who
can see each folder.
We need to clear the template-folder cache when adding a user to a service so
that we are not using out of date data about who can see each folder.
Added a folder permissions form to the page to invite users to services.
This only shows if the service has 'edit_folder_permissions' enabled,
and all folder checkboxes are checked by default. This change means that
InviteApiClient.create_invite now sends folder_permissions through to
notifications_api (so invites get created with folder permissions).
Started passing the folder_permissions through to notifications-api when
accepting an invite. This changes UserApiClient.add_user_to_service to
send folder_permissions to notifications_api so that new users get folder
permissions when they are added to the service.
The endpoint for adding a user to a service in api will now deal with
both user permissions and a user's folder permissions, so this changes
the format of the data we pass through.
The endpoint for setting permissions in api will now be used for both
user permissions and a user's folder permissions, so this changes the
format of the data we pass through.
when clients are defined in app/__init__.py, it increases the chance of
cyclical imports. By moving module level client singletons out to a
separate extensions file, we stop cyclical imports, but keep the same
code flow - the clients are still initialised in `create_app` in
`__init__.py`.
The redis client in particular is no longer separate - previously redis
was set up on the `NotifyAdminAPIClient` base class, but now there's one
singleton in `app.extensions`. This was done so that we can access redis
from outside of the existing clients.
This commit is the first step to disentangling the models from the API
clients. With the models in the same folder as the API clients it makes
it hard to import the API clients within the model without getting a
circular import.
After this commit the user API clients still has this problem, but at
least the service API client doesn’t.
Pytest is deprecating the direct calling of fixtures. One fixture that
we call directly quite a lot is `fake_uuid`. Since it just returns the
value of `sample_uuid()` we can either call that instead (where we need
a fixed value) or generate a new UUID each time (where a fixed value is
not needed).
we were seeing isort produce different outputs locally and in docker -
this was due to it having different opinions about whether the tests
module (ie all our unit tests) is a first party (local) or third party
(pip installed) import. It's a first party import, so by defining this
in the setup.cfg isort settings, we can force it to be consistent
between environments.
Note: I don't know why it was different in the first place though
Redis is giving us a big performance boost (it’s roughly halved the
median request time on the admin app).
Once we’re confident that it’s working properly[1] we can eke out a bit
more performance from it by keeping the caches alive for longer. As
far as I can tell we’re still using Redis in a very low-volume way[2],
so increasing the number of things we’re storing shouldn’t start taxing
our Redis server at all. But reducing the number of times we have to
hit the API to refresh the cache _should_ result in some performance
increase.
---
1. ie we’re not seeing instances of stale caches not being invalidated
2. We have 2.5G of available space in Redis. Here is our current usage:
```
used_memory:7728960
used_memory_human:7.37M
used_memory_rss:7728960
used_memory_peak:16563776
used_memory_peak_human:15.79M
used_memory_lua:37888
```
Accepting an invite changes:
- the `user_to_service` list of users returned by `GET /service/<id>`
- the `services` list return by `GET /user/<id>`
The latter change is causing the functional tests to fail.
In the same way, and for the same reasons that we’re caching the service
object.
Here’s a sample of the data returned by the API – so we should make sure
that any changes to this data invalidate the cache.
If we ever change a user’s phone number (for example) directly in the
database, then we will need to invalidate this cache manually.
```python
{
'data':{
'organisations':[
'4c707b81-4c6d-4d33-9376-17f0de6e0405'
],
'logged_in_at':'2018-04-10T11:41:03.781990Z',
'id':'2c45486e-177e-40b8-997d-5f4f81a461ca',
'email_address':'test@example.gov.uk',
'platform_admin':False,
'password_changed_at':'2018-01-01 10:10:10.100000',
'permissions':{
'42a9d4f2-1444-4e22-9133-52d9e406213f':[
'manage_api_keys',
'send_letters',
'manage_users',
'manage_templates',
'view_activity',
'send_texts',
'send_emails',
'manage_settings'
],
'a928eef8-0f25-41ca-b480-0447f29b2c20':[
'manage_users',
'manage_templates',
'manage_settings',
'send_texts',
'send_emails',
'send_letters',
'manage_api_keys',
'view_activity'
],
},
'state':'active',
'mobile_number':'07700900123',
'failed_login_count':0,
'name':'Example',
'services':[
'6078a8c0-52f5-4c4f-b724-d7d1ff2d3884',
'6afe3c1c-7fda-4d8d-aa8d-769c4bdf7803',
],
'current_session_id':'fea2ade1-db0a-4c90-93e7-c64a877ce83e',
'auth_type':'sms_auth'
}
}
```
in the db, we have several rows for single permissions - we separate
`send_messages` into `send_texts`, `send_emails` and `send_letters`,
and also `manage_service` into `manage_users` and `manage_settings`.
But on the front end we don't do anything with this distinction. It's
unhelpful for us to have to think about permissions as groups of things
when we can never split them up at all. So we should combine them. This
commit makes sure:
* when user models are read (from JSON direct from the API), we
should transform them from db permissions into roles.
* when permissions are persisted (editing permissions, and creating
invites), we should send db permissions to the API.
All other interaction with permissions (should just be the endpoint
decorator and checks in html templates generally) should use admin
roles.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
One of the things we need to know for a service to go live is whether
they have at least two users with the ‘manage service’ permission.
So this commit adds a method to the client to count how many users have
a given permission. We can do logic on this count later. But having the
counting done in the client feels like a cleaner separation of concerns.
Meant some refactoring of the way `service_id` is extracted from the
request, in order to make it easier to mock.
