When the broadcast service settings form is submitted it now removes all
permissions for users in notifications-api. This means it should be
clearing the user cache.
This naming was introduced in 2016 without explanation [1]. I find it
confusing because:
- It's reminiscent of "_app", which is a Python convention indicating
the variable is internal, so maybe avoid using it.
- It suggests there's some other "app" fixture I should be using (there
isn't, though).
The Python style guide describes using an underscore suffix to avoid
clashes with inbuilt names [1], which is sort of applicable if we need
to import the "app" module [2]. However, we can also avoid clashes by
choosing a different name, without the strange underscore.
[1]: 3b1d521c10
[2]: 78824f54fd/tests/app/main/views/test_forgot_password.py (L5)
Note, no option at the moment to set the service broadcast account type
as None, or back to without the broadcast permission. This has been done
for speed of development given the chance of us needing this is very
low. We can add it later if we need to.
Update all methods that were previous calling @cache.delete('service-{service-id}-template-None') to instead call _delete_template_cache_for_service
Remove call to get service templates, it's not needed since all template version cache is being deleted.
We were trying to delete the old 'template-{template-id}' keys but
should have been deleting the new keys which have the service id as part
of the key name. This was causing the cache to not be correctly purged
when we did things like update sender names or set defaults. This should
fix it.
We can drop use of the old key as we no longer need to read data from
the old key. Either data exists in the new key and we read it from there
or data doesn't exist in the new key and we go to the API to get it and
then set it in redis.
Note, the previous commit is important because it means we aren't at
risk of when this commit is being deployed out, of us getting stale data
from the old key.
We want to change cache keys for templates and broadcasts to include
their service ID. So cache keys should change from
`template-{template_id}-versions` to
`service-{service_id}-template-{template_id}-versions`.
The first step of this which needs to be deployed as a change first is
to delete both keys when updating service templates (even if they key is
not yet set). This means that when we release code in the next PR to
start setting the new key, we won't run into a case where either the old
or the new key can remain set with stale data.
sms sender, email reply to, letter contact blocks.
These are all cached within template, under `template.reply_to` - if the
template doesnt have a specific default, then that field stores the
service default. So when service default changes, we need to clear the
template cache so that it is updated to reflect that.
We already use this pattern for deleting the template cache for a bunch
of templates in `template_folder_api_client.move_to_folder`
The API now[1] accepts requests on `…/guest-list` as well as
`…/whitelist`. This commit starts using the former, which means:
- the use of ‘whitelist’ is fully gone from the admin app
- the API can stop using it, at least in URLs
1. As of https://github.com/alphagov/notifications-api/pull/2928
This should speed things up by:
- less time waiting for big blobs of JSON to come from Redis or the API
- less time spent deserialising big blobs of JSON
Fixes a bug where we were calling a wrapper method when instead we
should have been calling the redis_client. This had resulted in no
actual calls to redis happening.
When the admin app gets user objects from the API, these include a dict
of permissions by service for what the user can do to that services.
Permissions for inactive services are not included in the response as
per:
87cb6f2597/app/dao/permissions_dao.py (L66)
However, this causes a bug where a service is archived but cached user
data still tells us that the user has permissions to view the service.
This should not be the case and causes errors where users can still see
the archived service page, it's settings, and even request to go live
for it, because they are using old cached data for the user.
We solve this by deleting the users who are part of the service from the
cache.
We also delete the templates for this service from the cache as the
templates are also archived when we ask the API to archive the service
as per:
d95c0131e0/app/service/rest.py (L597)
Note, one decision I had to make was whether to delete the user cache
for just active team members or also invited users. Assuming an invited
user can't see the service until they've accepted their invite anyway, it
shouldn't make any difference whether we delete their cache or not.
We’re going to start using the returned letters summary to show some
info on the dashboard.
This means we will be accessing it more often than it changes. And we
know exactly when it changes because it’s us manually submitting the
references we get from DVLA.
This makes it a good candidate for being cached, and Redis is where we
cache stuff that we’d otherwise go to the API for.
Before, the delete decorator would delete the keys from Redis and then
we made the request to api to change the data. However, it is possible
that the cache could get re-populated in between these two things
happening, and so would cache outdated data.
This changes the order to send the api request first. We then always
delete the specified keys from Redis. Changing the order of the code in
the decorator changes the order in which the cache keys get deleted, so
the tests have been updated.
Added a folder permissions form to the page to invite users to services.
This only shows if the service has 'edit_folder_permissions' enabled,
and all folder checkboxes are checked by default. This change means that
InviteApiClient.create_invite now sends folder_permissions through to
notifications_api (so invites get created with folder permissions).
Started passing the folder_permissions through to notifications-api when
accepting an invite. This changes UserApiClient.add_user_to_service to
send folder_permissions to notifications_api so that new users get folder
permissions when they are added to the service.
when clients are defined in app/__init__.py, it increases the chance of
cyclical imports. By moving module level client singletons out to a
separate extensions file, we stop cyclical imports, but keep the same
code flow - the clients are still initialised in `create_app` in
`__init__.py`.
The redis client in particular is no longer separate - previously redis
was set up on the `NotifyAdminAPIClient` base class, but now there's one
singleton in `app.extensions`. This was done so that we can access redis
from outside of the existing clients.
Pytest is deprecating the direct calling of fixtures. One fixture that
we call directly quite a lot is `fake_uuid`. Since it just returns the
value of `sample_uuid()` we can either call that instead (where we need
a fixed value) or generate a new UUID each time (where a fixed value is
not needed).
Allows getting notification counts for a given number of days to
support services with custom data retention periods (admin
dashboard page should still display counts for the last 7 days,
while the notifications page displays all stored notifications).
we're not actually looking at the detailed service aspects - just
the stats. We're doing this in three places:
* dashboard
* notification activity page
* when checking jobs to see if we're over the daily limit
change these places to use a new api endpoint (service/id/statistics),
which hopefully be a little more performant, and will definitely be a
little more organised - moving away from generic endpoints with loads
of optional parameters.
We still need the detailed endpoints for the platform admin page tho.
Depends on https://github.com/alphagov/notifications-api/pull/1865
we were seeing isort produce different outputs locally and in docker -
this was due to it having different opinions about whether the tests
module (ie all our unit tests) is a first party (local) or third party
(pip installed) import. It's a first party import, so by defining this
in the setup.cfg isort settings, we can force it to be consistent
between environments.
Note: I don't know why it was different in the first place though
Redis is giving us a big performance boost (it’s roughly halved the
median request time on the admin app).
Once we’re confident that it’s working properly[1] we can eke out a bit
more performance from it by keeping the caches alive for longer. As
far as I can tell we’re still using Redis in a very low-volume way[2],
so increasing the number of things we’re storing shouldn’t start taxing
our Redis server at all. But reducing the number of times we have to
hit the API to refresh the cache _should_ result in some performance
increase.
---
1. ie we’re not seeing instances of stale caches not being invalidated
2. We have 2.5G of available space in Redis. Here is our current usage:
```
used_memory:7728960
used_memory_human:7.37M
used_memory_rss:7728960
used_memory_peak:16563776
used_memory_peak_human:15.79M
used_memory_lua:37888
```
This is easier to read than having to understand the arguments 1…n of
the cache decorator are ‘magic’, and gives us more flexibility about
how the cache keys are formatted, eg being able to add words in the
middle of them.
Also changes the key format for all templates to be
`service-{service_id}-templates` instead of `templates-{service_id}`
because then it’s clearer what the ID represents.
