We were trying to delete the old 'template-{template-id}' keys but
should have been deleting the new keys which have the service id as part
of the key name. This was causing the cache to not be correctly purged
when we did things like update sender names or set defaults. This should
fix it.
We can drop use of the old key as we no longer need to read data from
the old key. Either data exists in the new key and we read it from there
or data doesn't exist in the new key and we go to the API to get it and
then set it in redis.
Note, the previous commit is important because it means we aren't at
risk of when this commit is being deployed out, of us getting stale data
from the old key.
We want to change cache keys for templates and broadcasts to include
their service ID. So cache keys should change from
`template-{template_id}-versions` to
`service-{service_id}-template-{template_id}-versions`.
The first step of this which needs to be deployed as a change first is
to delete both keys when updating service templates (even if they key is
not yet set). This means that when we release code in the next PR to
start setting the new key, we won't run into a case where either the old
or the new key can remain set with stale data.
sms sender, email reply to, letter contact blocks.
These are all cached within template, under `template.reply_to` - if the
template doesnt have a specific default, then that field stores the
service default. So when service default changes, we need to clear the
template cache so that it is updated to reflect that.
We already use this pattern for deleting the template cache for a bunch
of templates in `template_folder_api_client.move_to_folder`
This should make the pages slightly quicker to load, because Redis will
return the JSON string faster than the API.
The only change that can happen to a broadcast which doesn’t go through
the admin app is a broadcast ending at its scheduled time. So this could
result in a cached broadcast having a status of `broadcasting` when it
had in fact finished. We already account for this here though:
b2b58ec044/app/models/broadcast_message.py (L89-L94)
The API now[1] accepts requests on `…/guest-list` as well as
`…/whitelist`. This commit starts using the former, which means:
- the use of ‘whitelist’ is fully gone from the admin app
- the API can stop using it, at least in URLs
1. As of https://github.com/alphagov/notifications-api/pull/2928
This should speed things up by:
- less time waiting for big blobs of JSON to come from Redis or the API
- less time spent deserialising big blobs of JSON
Fixes a bug where we were calling a wrapper method when instead we
should have been calling the redis_client. This had resulted in no
actual calls to redis happening.
When the admin app gets user objects from the API, these include a dict
of permissions by service for what the user can do to that services.
Permissions for inactive services are not included in the response as
per:
87cb6f2597/app/dao/permissions_dao.py (L66)
However, this causes a bug where a service is archived but cached user
data still tells us that the user has permissions to view the service.
This should not be the case and causes errors where users can still see
the archived service page, it's settings, and even request to go live
for it, because they are using old cached data for the user.
We solve this by deleting the users who are part of the service from the
cache.
We also delete the templates for this service from the cache as the
templates are also archived when we ask the API to archive the service
as per:
d95c0131e0/app/service/rest.py (L597)
Note, one decision I had to make was whether to delete the user cache
for just active team members or also invited users. Assuming an invited
user can't see the service until they've accepted their invite anyway, it
shouldn't make any difference whether we delete their cache or not.
We’re going to start using the returned letters summary to show some
info on the dashboard.
This means we will be accessing it more often than it changes. And we
know exactly when it changes because it’s us manually submitting the
references we get from DVLA.
This makes it a good candidate for being cached, and Redis is where we
cache stuff that we’d otherwise go to the API for.
As per https://www.pivotaltracker.com/story/show/170796514 we want to make the delete template confirmation dialog box more consistent and clear.
The API has been updated with a new endpoint that only returns the last-used date, this date is more accurate since it goes to the ft_notification_status table, if the notification table is empty.
The API clients should just deal with calling the API and returning the
data from it.
Inferring things from the data is more logically done at the model
layer. But we couldn’t do that before, because we didn’t have a model
layer for jobs.
Now persisting the address to the "to" field of the Notification, after the notification has been validated.
If the letter is pending validation, then "Checking..." will appear as the identifier for the letter.
If the letter has passed validation, then the first line of the address (now persisted in the "to" field) will be displayed, with the client reference underneath.
If the letter has failed validation the "Provided as PDF" will show be displayed, which is now the initial value of the "to" field.
We were using user fixtures in a lot of parameterized tests, but this is
no longer allowed in Pytest 5. To avoid having to split up the parametrized
tests (which would make the test files a lot longer and slightly more
difficult to read) this commit creates functions which return various types
of user json so that we can use these as the test parameters instead.
We hardcode this as second class for the moment but eventually
will let the user pick.
Currently the API appears to do no validation, e.g. a json
schema, that rejects API calls with the extra key for postage.
Next steps will be to put a PR into the API that will expect a
postage value in the request and save it with the rest of the
notification. Then when that is done we can add the user interface
to the admin app to let the user pick the postage.
Added a send button which only appears on the page if the query string
indicates that the PDF is valid. Before actually sending, we check that
the service has the right permissions and that the metadata for the
letter confirms the letter is valid (because the query string can be
changed).
Before, the delete decorator would delete the keys from Redis and then
we made the request to api to change the data. However, it is possible
that the cache could get re-populated in between these two things
happening, and so would cache outdated data.
This changes the order to send the api request first. We then always
delete the specified keys from Redis. Changing the order of the code in
the decorator changes the order in which the cache keys get deleted, so
the tests have been updated.
This report will be used by the engagement team. There is a form to give
a start and end date for the report, and the form is then downloaded
as a CSV file when the form is submitted.
This should make the ‘All organisations’ page load a lil’ bit quicker.
Still worth caching the domains separately so the response is smaller
when we only care about domains. This is because the code that uses the
domains is part of the sign up flow, so it’s really important that it’s
snappy.
Users can only be archived by Platform Admin from the user page
(/users/<user_id>). This removes them from all services and orgs and
updates their details.
The data flow of other bits of our application looks like this:
```
API (returns JSON)
⬇
API client (returns a built in type, usually `dict`)
⬇
Model (returns an instance, eg of type `Service`)
⬇
View (returns HTML)
```
The user API client was architected weirdly, in that it returned a model
directly, like this:
```
API (returns JSON)
⬇
API client (returns a model, of type `User`, `InvitedUser`, etc)
⬇
View (returns HTML)
```
This mixing of different layers of the application is bad because it
makes it hard to write model code that doesn’t have circular
dependencies. As our application gets more complicated we will be
relying more on models to manage this complexity, so we should make it
easy, not hard to write them.
It also means that most of our mocking was of the User model, not just
the underlying JSON. So it would have been easy to introduce subtle bugs
to the user model, because it wasn’t being comprehensively tested. A lot
of the changed lines of code in this commit mean changing the tests to
mock only the JSON, which means that the model layer gets implicitly
tested.
For those reasons this commit changes the user API client to return
JSON, not an instance of `User` or other models.
The domains lookup is a bit slow because it’s serialising all the
organisations in the database. Since we’re putting this in the sign up
flow it should feel snappy, so lets cache just the domain bit of it in
Redis.
We’re deprecating storing the domain as text on a branding in favour of
a database relationship between branding and organisation.
We need to do this now in order to remove the validation on these fields
(which depends on the data in `domains.yml`)
The api endpoint to get all template folders also returns the users who
can see each folder.
We need to clear the template-folder cache when adding a user to a service so
that we are not using out of date data about who can see each folder.
Added a folder permissions form to the page to invite users to services.
This only shows if the service has 'edit_folder_permissions' enabled,
and all folder checkboxes are checked by default. This change means that
InviteApiClient.create_invite now sends folder_permissions through to
notifications_api (so invites get created with folder permissions).
Started passing the folder_permissions through to notifications-api when
accepting an invite. This changes UserApiClient.add_user_to_service to
send folder_permissions to notifications_api so that new users get folder
permissions when they are added to the service.
We were already invitializing InvitedUser with folder_permissions
(defaulting to None), but this removes the default and adds
folder_permissions to the serialize method. Folder permissions should
now always be returned from api, either as an empty list or a list of
UUIDs.
The endpoint for adding a user to a service in api will now deal with
both user permissions and a user's folder permissions, so this changes
the format of the data we pass through.
The endpoint for setting permissions in api will now be used for both
user permissions and a user's folder permissions, so this changes the
format of the data we pass through.
when clients are defined in app/__init__.py, it increases the chance of
cyclical imports. By moving module level client singletons out to a
separate extensions file, we stop cyclical imports, but keep the same
code flow - the clients are still initialised in `create_app` in
`__init__.py`.
The redis client in particular is no longer separate - previously redis
was set up on the `NotifyAdminAPIClient` base class, but now there's one
singleton in `app.extensions`. This was done so that we can access redis
from outside of the existing clients.
This is a platform admin page which lists all letter brands and their
domain. Once the page to edit a letter brand has been created, each
brand will link to its edit page, but for now this is just a list of
brand names and domains.
