For both SMS senders and email reply to addresses this commit adds:
- a delete link
- a confirmation loop
It doesn’t let users delete:
- default SMS senders or reply to addresses (they always have to have
one)
- inbound numbers
It assumes that the API will allow updating of an attribute named
`active` on the respective database rows. It could work in a different
way. We can’t do complete deletion though because these will still be
keyed to notifications.
Redis is giving us a big performance boost (it’s roughly halved the
median request time on the admin app).
Once we’re confident that it’s working properly[1] we can eke out a bit
more performance from it by keeping the caches alive for longer. As
far as I can tell we’re still using Redis in a very low-volume way[2],
so increasing the number of things we’re storing shouldn’t start taxing
our Redis server at all. But reducing the number of times we have to
hit the API to refresh the cache _should_ result in some performance
increase.
---
1. ie we’re not seeing instances of stale caches not being invalidated
2. We have 2.5G of available space in Redis. Here is our current usage:
```
used_memory:7728960
used_memory_human:7.37M
used_memory_rss:7728960
used_memory_peak:16563776
used_memory_peak_human:15.79M
used_memory_lua:37888
```
This is easier to read than having to understand the arguments 1…n of
the cache decorator are ‘magic’, and gives us more flexibility about
how the cache keys are formatted, eg being able to add words in the
middle of them.
Also changes the key format for all templates to be
`service-{service_id}-templates` instead of `templates-{service_id}`
because then it’s clearer what the ID represents.
A lot of the frequently-used pages in the admin app rely on the API to
get templates.
So this commit adds three new caches:
- a single template version (including a key without a version number,
which is the current version)
- all the templates for a service
- all versions of a template
The first will be the most crucial for performance, but there’s not much
cost to adding the other two.
`@cache.delete('user', 'user_id')` is easier to read and understand than
`@cache.delete('user', key_from_args=[1])`. This will become even more
apparent if we have to start doing stuff like `key_from_args=[1, 5]`,
which is a lot more opaque than just saying
`'service_id', 'template_id'`.
It does make the implementation a bit more complex, but I’m not too
worried about that because:
- the tests are solid
- it’s nicely encapsulated
This `*params` argument seems to be copy/pasted boilerplate. It’s not
used by any consumers of this client, and makes it harder to write a
decorator for this function.
Accepting an invite changes:
- the `user_to_service` list of users returned by `GET /service/<id>`
- the `services` list return by `GET /user/<id>`
The latter change is causing the functional tests to fail.
In the same way, and for the same reasons that we’re caching the service
object.
Here’s a sample of the data returned by the API – so we should make sure
that any changes to this data invalidate the cache.
If we ever change a user’s phone number (for example) directly in the
database, then we will need to invalidate this cache manually.
```python
{
'data':{
'organisations':[
'4c707b81-4c6d-4d33-9376-17f0de6e0405'
],
'logged_in_at':'2018-04-10T11:41:03.781990Z',
'id':'2c45486e-177e-40b8-997d-5f4f81a461ca',
'email_address':'test@example.gov.uk',
'platform_admin':False,
'password_changed_at':'2018-01-01 10:10:10.100000',
'permissions':{
'42a9d4f2-1444-4e22-9133-52d9e406213f':[
'manage_api_keys',
'send_letters',
'manage_users',
'manage_templates',
'view_activity',
'send_texts',
'send_emails',
'manage_settings'
],
'a928eef8-0f25-41ca-b480-0447f29b2c20':[
'manage_users',
'manage_templates',
'manage_settings',
'send_texts',
'send_emails',
'send_letters',
'manage_api_keys',
'view_activity'
],
},
'state':'active',
'mobile_number':'07700900123',
'failed_login_count':0,
'name':'Example',
'services':[
'6078a8c0-52f5-4c4f-b724-d7d1ff2d3884',
'6afe3c1c-7fda-4d8d-aa8d-769c4bdf7803',
],
'current_session_id':'fea2ade1-db0a-4c90-93e7-c64a877ce83e',
'auth_type':'sms_auth'
}
}
```
Most of the time spent by the admin app to generate a page is spent
waiting for the API. This is slow for three reasons:
1. Talking to the API means going out to the internet, then through
nginx, the Flask app, SQLAlchemy, down to the database, and then
serialising the result to JSON and making it into a HTTP response
2. Each call to the API is synchronous, therefore if a page needs 3 API
calls to render then the second API call won’t be made until the
first has finished, and the third won’t start until the second has
finished
3. Every request for a service page in the admin app makes a minimum
of two requests to the API (`GET /service/…` and `GET /user/…`)
Hitting the database will always be the slowest part of an app like
Notify. But this slowness is exacerbated by 2. and 3. Conversely every
speedup made to 1. is multiplied by 2. and 3.
So this pull request aims to make 1. a _lot_ faster by taking nginx,
Flask, SQLAlchemy and the database out of the equation. It replaces them
with Redis, which as an in-memory key/value store is a lot faster than
Postgres. There is still the overhead of going across the network to
talk to Redis, but the net improvement is vast.
This commit only caches the `GET /service` response, but is written in
such a way that we can easily expand to caching other responses down the
line.
The tradeoff here is that our code is more complex, and we risk
introducing edge cases where a cache becomes stale. The mitigations
against this are:
- invalidating all caches after 24h so a stale cache doesn’t remain
around indefinitely
- being careful when we add new stuff to the service response
---
Some indicative numbers, based on:
- `GET http://localhost:6012/services/<service_id>/template/<template_id>`
- with the admin app running locally
- talking to Redis running locally
- also talking to the API running locally, itself talking to a local
Postgres instance
- times measured with Chrome web inspector, average of 10 requests
╲ | No cache | Cache service | Cache service and user | Cache service, user and template
-- | -- | -- | -- | --
**Request time** | 136ms | 97ms | 73ms | 37ms
**Improvement** | 0% | 41% | 88% | 265%
---
Estimates of how much storage this requires:
- Services: 1,942 on production × 2kb = 4Mb
- Users: 4,534 on production × 2kb = 9Mb
- Templates: 7,079 on production × 4kb = 28Mb
it was only used by the choose service page, and then only in kludgy
ways (eg: creating a list containing one item called "add service"),
so lets rip it out and make this page bespoke. Especially now that it's
changed so much.
this endpoint should probably only be used for the choose-service page
also create an OrganisationBrowsableItem to aid rendering of them
in the front-end.
We were counting users who had the `manage_settings` permission. This
is the old name for it, therefore there would never be any users with
this permission, so the tick would never go green.
The new name for the permission is `manage_service`. This commit fixes
the error, and adds an extra safeguard against something like this
happening again.
* Moved the notifications code to go to admin to get the the template
preview document rather than go to template preview.
This will remove the logic from admin and place it in api so it is
easier to expand on later when there are precompiled PDFs
* Added some error handling if API returns an API error.
Caught the error and displayed an error PNG so it is obvious something
failed. Currently it displayed a thumbnail of a png over the top of the
loading page, and therefore it wasn't obvious of the state.
view args are parameters within the route. for example,
`/organisation/<org_id>/users`. If there is an org_id, then check that
the user is part of that organisation (users.organisations is a list of
all orgs that user is a member of).
* platform admins ignore this check if restrict_admin_usage=False
* if an endpoint has both org_id and service_id, org_id takes
precedence, but we should probably revisit this if we ever need
to create such an endpoint.
* you now call `@user_has_permissions()` with no arguments for
organisation endpoints - we can look at this if we decide we want
more clarity.
* you should never call user_has_permissions without any arguments
for endpoints that aren't organisation-based. We'll raise
NotImplementedError if you do.
we branch on any_ to either say "require ALL these permissions" or
"require ANY of these permissions". But we only ever call the decorator
with one permission, or with any_=True, so it's unnecessary
rather than allow admins to do everything specifically, we should
only block them from things we conciously don't want them to do.
This is "Don't let platform admins send letters from services they're
not in". Everything else the platform admins can do.
This is step one, adding a restrict_admin_usage flag, and setting that
for those restricted endpoints around creating api keys, uploading CSVs
and sending one-off messages.
Also, this commit separates the two use cases for permissions:
* user.has_permission for access control
* user.has_permission_for_service for user info - this is used for
showing checkboxes on the manage-users page for example
With this, we can remove the admin_override flag from the permission
decorator.
in the db, we have several rows for single permissions - we separate
`send_messages` into `send_texts`, `send_emails` and `send_letters`,
and also `manage_service` into `manage_users` and `manage_settings`.
But on the front end we don't do anything with this distinction. It's
unhelpful for us to have to think about permissions as groups of things
when we can never split them up at all. So we should combine them. This
commit makes sure:
* when user models are read (from JSON direct from the API), we
should transform them from db permissions into roles.
* when permissions are persisted (editing permissions, and creating
invites), we should send db permissions to the API.
All other interaction with permissions (should just be the endpoint
decorator and checks in html templates generally) should use admin
roles.
when added to a service, all users are given the view_activity
permission. So, if that's included in the list, we don't need `any_`,
and we don't need any of the other permissions.
preview document rather than go to template preview.
This will remove the logic from admin and place it in api so it is
easier to expand on later when there are precompiled PDFs
Currently requests to the API made from the admin app are going from
PaaS admin app to the nginx router ELB, which then routes them back
to the api app on PaaS.
This makes sense for external requests, but for requests made from
the admin app we could skip nginx and go directly to the api PaaS
host, which should reduce load on the nginx instances and
potentially reduce latency of the api requests.
API apps on PaaS are checking the X-Custom-Forwarder header (which
is set by nginx on proxy_pass requests) to only allow requests going
through the proxy.
This adds the custom header to the API client requests, so that they
can pass that header check without going through nginx.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
One of the things we need to know for a service to go live is whether
they have at least two users with the ‘manage service’ permission.
So this commit adds a method to the client to count how many users have
a given permission. We can do logic on this count later. But having the
counting done in the client feels like a cleaner separation of concerns.
Meant some refactoring of the way `service_id` is extracted from the
request, in order to make it easier to mock.
When users request to go live we check stuff like:
- if they’ve added templates
- if they have email templates (then we can check their reply to
address)
This commit adds a method to do this programatically rather than
manually.
We _could_ do this in SQL, but for page that’s used intermittently it
doesn’t feel worth the work/optimisation (and the client method is at
least in place now if we do ever need to lean on this code more
heavily).
Notifications-api now needs the service_id to check the uniqueness of
the service name when trying to change it. This is to allow a user to
successfully change the pluralization or the case of their service name.
