One of the big things we found in user research was that people were
uncertain what the effect of giving someone basic view was.
So in the spirit of ‘show don’t tell’, this commit adds a way for users
to preview basic view. They can go into the preview and click around as
much as they like, just as if they really had the basic view assigned to
them.
Once they have seen enough they can return to the settings page where
they can decide whether or not to switch basic view on for real.
This commit changes the form that the user sees when inviting or editing
another user, if the service has the ‘caseworking’ permission set.
This will allow creating a new type of user, one who only has the
`send_messages` permission, without the `view_activity` permission.
We are doing this because we think there are a number of services with a
lot of users who don’t need to see the dashboard, or the other team
members, and that we can make a simpler interface for these users.
We’ve had a user who’s said:
> Seems configured callbacks cannot be removed once they’re set as the
> fields have a presence check. Is that intentional?
This means it’s not working as they expect. Rather than have to go and
change stuff in the database for them, let’s make it work as they’d
expect.
Only lets you clear the form if you remove both the token and the URL.
In API, the endpoint for the new platform admin stats page has been
moved to a platform stats blueprint. This means we now need a platform
stats client.
* Added a new method to the ComplaintApiClient to get the total
complaints by date range from the API.
* Added a new method to the ServiceAPIClient to get the new platform
admin stats data from the API.
Added a page which lets users with the 'manage_service' permission change the
contact link for their service. There are no links to this page yet
since only services using document download will need to set a contact
link.
A new platform admin page Email complaints has been added to surface those complaints.
Eventually the complaints will be visible to the services so they can remove the email address from their mailing list.
Next thing to implement is "x email complaints" warning on the platform admin summary page.
we're not actually looking at the detailed service aspects - just
the stats. We're doing this in three places:
* dashboard
* notification activity page
* when checking jobs to see if we're over the daily limit
change these places to use a new api endpoint (service/id/statistics),
which hopefully be a little more performant, and will definitely be a
little more organised - moving away from generic endpoints with loads
of optional parameters.
We still need the detailed endpoints for the platform admin page tho.
Depends on https://github.com/alphagov/notifications-api/pull/1865
For both SMS senders and email reply to addresses this commit adds:
- a delete link
- a confirmation loop
It doesn’t let users delete:
- default SMS senders or reply to addresses (they always have to have
one)
- inbound numbers
It assumes that the API will allow updating of an attribute named
`active` on the respective database rows. It could work in a different
way. We can’t do complete deletion though because these will still be
keyed to notifications.
Redis is giving us a big performance boost (it’s roughly halved the
median request time on the admin app).
Once we’re confident that it’s working properly[1] we can eke out a bit
more performance from it by keeping the caches alive for longer. As
far as I can tell we’re still using Redis in a very low-volume way[2],
so increasing the number of things we’re storing shouldn’t start taxing
our Redis server at all. But reducing the number of times we have to
hit the API to refresh the cache _should_ result in some performance
increase.
---
1. ie we’re not seeing instances of stale caches not being invalidated
2. We have 2.5G of available space in Redis. Here is our current usage:
```
used_memory:7728960
used_memory_human:7.37M
used_memory_rss:7728960
used_memory_peak:16563776
used_memory_peak_human:15.79M
used_memory_lua:37888
```
This is easier to read than having to understand the arguments 1…n of
the cache decorator are ‘magic’, and gives us more flexibility about
how the cache keys are formatted, eg being able to add words in the
middle of them.
Also changes the key format for all templates to be
`service-{service_id}-templates` instead of `templates-{service_id}`
because then it’s clearer what the ID represents.
A lot of the frequently-used pages in the admin app rely on the API to
get templates.
So this commit adds three new caches:
- a single template version (including a key without a version number,
which is the current version)
- all the templates for a service
- all versions of a template
The first will be the most crucial for performance, but there’s not much
cost to adding the other two.
`@cache.delete('user', 'user_id')` is easier to read and understand than
`@cache.delete('user', key_from_args=[1])`. This will become even more
apparent if we have to start doing stuff like `key_from_args=[1, 5]`,
which is a lot more opaque than just saying
`'service_id', 'template_id'`.
It does make the implementation a bit more complex, but I’m not too
worried about that because:
- the tests are solid
- it’s nicely encapsulated
This `*params` argument seems to be copy/pasted boilerplate. It’s not
used by any consumers of this client, and makes it harder to write a
decorator for this function.
Accepting an invite changes:
- the `user_to_service` list of users returned by `GET /service/<id>`
- the `services` list return by `GET /user/<id>`
The latter change is causing the functional tests to fail.
In the same way, and for the same reasons that we’re caching the service
object.
Here’s a sample of the data returned by the API – so we should make sure
that any changes to this data invalidate the cache.
If we ever change a user’s phone number (for example) directly in the
database, then we will need to invalidate this cache manually.
```python
{
'data':{
'organisations':[
'4c707b81-4c6d-4d33-9376-17f0de6e0405'
],
'logged_in_at':'2018-04-10T11:41:03.781990Z',
'id':'2c45486e-177e-40b8-997d-5f4f81a461ca',
'email_address':'test@example.gov.uk',
'platform_admin':False,
'password_changed_at':'2018-01-01 10:10:10.100000',
'permissions':{
'42a9d4f2-1444-4e22-9133-52d9e406213f':[
'manage_api_keys',
'send_letters',
'manage_users',
'manage_templates',
'view_activity',
'send_texts',
'send_emails',
'manage_settings'
],
'a928eef8-0f25-41ca-b480-0447f29b2c20':[
'manage_users',
'manage_templates',
'manage_settings',
'send_texts',
'send_emails',
'send_letters',
'manage_api_keys',
'view_activity'
],
},
'state':'active',
'mobile_number':'07700900123',
'failed_login_count':0,
'name':'Example',
'services':[
'6078a8c0-52f5-4c4f-b724-d7d1ff2d3884',
'6afe3c1c-7fda-4d8d-aa8d-769c4bdf7803',
],
'current_session_id':'fea2ade1-db0a-4c90-93e7-c64a877ce83e',
'auth_type':'sms_auth'
}
}
```
Most of the time spent by the admin app to generate a page is spent
waiting for the API. This is slow for three reasons:
1. Talking to the API means going out to the internet, then through
nginx, the Flask app, SQLAlchemy, down to the database, and then
serialising the result to JSON and making it into a HTTP response
2. Each call to the API is synchronous, therefore if a page needs 3 API
calls to render then the second API call won’t be made until the
first has finished, and the third won’t start until the second has
finished
3. Every request for a service page in the admin app makes a minimum
of two requests to the API (`GET /service/…` and `GET /user/…`)
Hitting the database will always be the slowest part of an app like
Notify. But this slowness is exacerbated by 2. and 3. Conversely every
speedup made to 1. is multiplied by 2. and 3.
So this pull request aims to make 1. a _lot_ faster by taking nginx,
Flask, SQLAlchemy and the database out of the equation. It replaces them
with Redis, which as an in-memory key/value store is a lot faster than
Postgres. There is still the overhead of going across the network to
talk to Redis, but the net improvement is vast.
This commit only caches the `GET /service` response, but is written in
such a way that we can easily expand to caching other responses down the
line.
The tradeoff here is that our code is more complex, and we risk
introducing edge cases where a cache becomes stale. The mitigations
against this are:
- invalidating all caches after 24h so a stale cache doesn’t remain
around indefinitely
- being careful when we add new stuff to the service response
---
Some indicative numbers, based on:
- `GET http://localhost:6012/services/<service_id>/template/<template_id>`
- with the admin app running locally
- talking to Redis running locally
- also talking to the API running locally, itself talking to a local
Postgres instance
- times measured with Chrome web inspector, average of 10 requests
╲ | No cache | Cache service | Cache service and user | Cache service, user and template
-- | -- | -- | -- | --
**Request time** | 136ms | 97ms | 73ms | 37ms
**Improvement** | 0% | 41% | 88% | 265%
---
Estimates of how much storage this requires:
- Services: 1,942 on production × 2kb = 4Mb
- Users: 4,534 on production × 2kb = 9Mb
- Templates: 7,079 on production × 4kb = 28Mb
it was only used by the choose service page, and then only in kludgy
ways (eg: creating a list containing one item called "add service"),
so lets rip it out and make this page bespoke. Especially now that it's
changed so much.
this endpoint should probably only be used for the choose-service page
also create an OrganisationBrowsableItem to aid rendering of them
in the front-end.
We were counting users who had the `manage_settings` permission. This
is the old name for it, therefore there would never be any users with
this permission, so the tick would never go green.
The new name for the permission is `manage_service`. This commit fixes
the error, and adds an extra safeguard against something like this
happening again.
* Moved the notifications code to go to admin to get the the template
preview document rather than go to template preview.
This will remove the logic from admin and place it in api so it is
easier to expand on later when there are precompiled PDFs
* Added some error handling if API returns an API error.
Caught the error and displayed an error PNG so it is obvious something
failed. Currently it displayed a thumbnail of a png over the top of the
loading page, and therefore it wasn't obvious of the state.
view args are parameters within the route. for example,
`/organisation/<org_id>/users`. If there is an org_id, then check that
the user is part of that organisation (users.organisations is a list of
all orgs that user is a member of).
* platform admins ignore this check if restrict_admin_usage=False
* if an endpoint has both org_id and service_id, org_id takes
precedence, but we should probably revisit this if we ever need
to create such an endpoint.
* you now call `@user_has_permissions()` with no arguments for
organisation endpoints - we can look at this if we decide we want
more clarity.
* you should never call user_has_permissions without any arguments
for endpoints that aren't organisation-based. We'll raise
NotImplementedError if you do.
we branch on any_ to either say "require ALL these permissions" or
"require ANY of these permissions". But we only ever call the decorator
with one permission, or with any_=True, so it's unnecessary
rather than allow admins to do everything specifically, we should
only block them from things we conciously don't want them to do.
This is "Don't let platform admins send letters from services they're
not in". Everything else the platform admins can do.
This is step one, adding a restrict_admin_usage flag, and setting that
for those restricted endpoints around creating api keys, uploading CSVs
and sending one-off messages.
Also, this commit separates the two use cases for permissions:
* user.has_permission for access control
* user.has_permission_for_service for user info - this is used for
showing checkboxes on the manage-users page for example
With this, we can remove the admin_override flag from the permission
decorator.
in the db, we have several rows for single permissions - we separate
`send_messages` into `send_texts`, `send_emails` and `send_letters`,
and also `manage_service` into `manage_users` and `manage_settings`.
But on the front end we don't do anything with this distinction. It's
unhelpful for us to have to think about permissions as groups of things
when we can never split them up at all. So we should combine them. This
commit makes sure:
* when user models are read (from JSON direct from the API), we
should transform them from db permissions into roles.
* when permissions are persisted (editing permissions, and creating
invites), we should send db permissions to the API.
All other interaction with permissions (should just be the endpoint
decorator and checks in html templates generally) should use admin
roles.
when added to a service, all users are given the view_activity
permission. So, if that's included in the list, we don't need `any_`,
and we don't need any of the other permissions.
preview document rather than go to template preview.
This will remove the logic from admin and place it in api so it is
easier to expand on later when there are precompiled PDFs
Currently requests to the API made from the admin app are going from
PaaS admin app to the nginx router ELB, which then routes them back
to the api app on PaaS.
This makes sense for external requests, but for requests made from
the admin app we could skip nginx and go directly to the api PaaS
host, which should reduce load on the nginx instances and
potentially reduce latency of the api requests.
API apps on PaaS are checking the X-Custom-Forwarder header (which
is set by nginx on proxy_pass requests) to only allow requests going
through the proxy.
This adds the custom header to the API client requests, so that they
can pass that header check without going through nginx.
