API gives an error if it tries to add a user to a service and that user is
already a member of the service. This situation shouldn't occur - admin checks
if an invited user is a member of a service before calling API, but we
have seen this error occurring when there are two requests processing at
the same time.
This change catches the errors from API if a user is already a member of
a service and redirects the user to the service dashboard so that they
don't see an error page.
Some teams have started uploading quite a lot of letters (in the
hundreds per week). They’re also uploading CSVs of emails. This means
the uploads page ends up quite jumbled.
This is because:
- there’s just a lot of items to scan through
- conceptually it’s a bit odd to have batches of things displayed
alongside individual things on the same page
So instead we’re going to start grouping together uploaded letters. This
will be by the date on which we ‘start’ printing them, or in other
words the time at which they can no longer be cancelled.
This feels like a natural grouping, and it matches what we know about
people’s mental models of ‘batches’ and ‘runs’ when talking about
printing.
This grouping will be done in the API, so all this commit need to do is:
- be ready to display this new type of pseudo-job
- link to the page that displays all the uploaded letters for a given
print day
For services with permission, they can now put international addresses
into their spreadsheets without getting a postcode error.
This also means they can start using address line 7 instead of postcode,
since it doesn’t make sense to put a country in a field called
‘postcode’. But this will be undocumented to start with, because we’re
not giving any real users the permission.
It does now mean that the number of possible placeholders (7 + postcode)
is greater than the number of allowed placeholders (7), so we have to
account for that in the one-off address flow where we’re populating the
placeholders automatically. We’re sticking with 6 + postcode here for
backwards compatibility.
Our rules about address columns are relaxing, so that none of them are
mandatory any more. Instead you just need any 3 of the 7 to make a valid
address.
This commit updates our error messaging to reflect that.
`dir(object)` is a useful Python function that tells you what attributes
and methods an object has. It’s also used by tools like iPython and IDEs
for code completion.
Some of the attributes of a `JSONModel` are dynamic, based on what
fields we expect in the underlying JSON. Therefore they don’t
automatically end up in the result of calling `dir`. To get around this
we can implement our own `__dir__` method, which also returns the names
of the fields we’re expecting the the JSON.
Inspired by this Raymond Hettinger tweet:
> #python tip: If you add attributes to an API with __getattr__() or
> __getattribute__(), remember to update __dir__() to make the extension
> introspectable.
— https://twitter.com/raymondh/status/1249860863525146624
We’re caching the organisation name, but still talking to the API
to see if the organisation exists.
`Service().organisation_id` only goes to the JSON for the service.
`Service().organisation` makes a separate API call.
We only need the former to know if a service belongs to an organisation.
A lot of pages in the admin app are now generated entirely from Redis,
without touching the API.
The one remaining API call that a lot of pages make, when the user is
platform admin or a member of an organisation, is to get the name of
the current service’s organisation.
This commit adds some code to start caching that as well, which should
speed up page load times for when we’re clicking around the admin app
(it’s typically 100ms just to get the organisation, and more than that
when the API is under load).
This means changing the service model to get the organisation from the
API by ID, not by service ID. Otherwise it would be very hard to clear
the cache if the name of the organisation ever changed.
We can’t cache the whole organisation because it has a
`count_of_live_services` field which can change at any time, without an
update being made.
This is for tickets coming from non-logged-in users. It’s effectively
the same as reporting a problem, but doesn’t have the banner about
the status page (because we can’t tell if they’re actually reporting a
problem now we’re not asking).
It also gives a more generic page title.
When we first built the contact list feature this endpoint didn’t exist.
Now it does we can remove the slightly cludgy looping-through-all-the-lists
code.
On the uploads page this should be newest first, same as the scheduled
and immediate jobs on this page.
On the _choose_ page this should be alphabetical, same as choosing a
template.
Uploads page is where all the stuff you’ve uploaded lives. Now you can
upload contact lists they should live here too.
They always come first because they’re the most-removed from stuff
you’ve sent.
You can’t send an email message template to a list of phone numbers. So
we shouldn’t show you the lists of phone numbers when you’ve chosen an
email template.
You’ll be able to use a contact list by first choosing a template, then
choosing the list you want to use.
At the moment this shows all lists, not just the ones that are
compatible with your template.
In order to use a contact list we’re going to put you in the normal
upload a spreadsheet journey. Except without having to upload again.
So this commit adds an endpoint that takes the file from the contact
list bucket, puts it in the same place it would have gone if you’d
uploaded it, then sends you on your way.
We don’t want to muddy them up with the normal CSV uploads.
I’ve tried to reuse the existing S3 code where possible because it’s
well tested.
Buckets have already been created.
We increasingly have teams wanting to do business-continuity type
messaging. They might be without access to their normal systems, which
is where they would otherwise go to get the list of email addresses or
phone numbers.
So we want to give them a place in Notify where they can store their
spreadsheets and use them at a later date.
For the initial pass we’re going to scope this to only allowing
spreadsheets with one column, ie just phone numbers/email addresses.
This is because:
- it minimises the amount of personal info we’re storing
- it reduces the chance of getting a placeholder error when you go to
send the message, which is probably a high-stress situation where you
might not be able to re-generate the file
The code for this is mostly copied from the existing upload CSV journey.
It’s quite duplicative, but that’s what I needed to do to get this out
quickly. There are opportunities for refactoring later.
Similarly, I would have liked to split this up into better commit
messages, but it really was a case of just bashing code out until it
worked 😳
This commit does not:
- implement the ‘view a contact list page’ (it just has a placeholder
because the API isn’t ready at the moment)
- link to this page (because it’s not ready to use yet)
flask_login sets a bunch of variables in the session object. We only use
one of them, `user_id`. We set that to the user id from the database,
and refer to it all over the place.
However, in flask_login 0.5.0 they prefix this with an underscore to
prevent people accidentally overwriting it etc. So when a user logs in
we need to make sure that we set user_id manually so we can still use
it.
flask_login sets a bunch of variables on the `flask.session` object.
However, this session object isn't the one that gets passed in to the
request context by flask - that one can only be modified outside of
requests from within the session_transaction context manager (see [1]).
So, flask_login populates the normal session and then we need to copy
all of those values across.
We didn't need to do this previously because we already set the
`user_id` value on line 20 of tests/__init__.py, but now that
flask_login is looking for `_user_id` instead we need to do this
properly.
[1] https://flask.palletsprojects.com/en/1.1.x/testing/#accessing-and-modifying-sessions
flask_login sets a bunch of variables in the session object. We only use
one of them, `user_id`. We set that to the user id from the database,
and refer to it all over the place.
However, in flask_login 0.5.0 they prefix this with an underscore to
prevent people accidentally overwriting it etc. So when a user logs in
we need to make sure that we set user_id manually so we can still use
it.
flask_login sets a bunch of variables on the `flask.session` object.
However, this session object isn't the one that gets passed in to the
request context by flask - that one can only be modified outside of
requests from within the session_transaction context manager (see [1]).
So, flask_login populates the normal session and then we need to copy
all of those values across.
We didn't need to do this previously because we already set the
`user_id` value on line 20 of tests/__init__.py, but now that
flask_login is looking for `_user_id` instead we need to do this
properly.
[1] https://flask.palletsprojects.com/en/1.1.x/testing/#accessing-and-modifying-sessions
Because ModelList implements `__add__` we can do the following:
```python
ImmediateJobs() + ScheduledJobs()
ImmediateJobs() + []
```
Both of these call the `__add__` method of `ImmediateJobs`.
What we can’t do is this:
```python
[] + ScheduledJobs()
```
That tries to call the `__add__` method of list, which doesn’t know what
to do with an instance of `ModelList`.
The Pythonic way to deal with this is to implement `__radd__` (right
add) which is invoked when our instance is on the right hand side of the
addition operator.
I’m hoping that if I can design something that clearly differentiates
them then we won’t need to do so by putting them in separate tables,
which then need labelling, which would clutter up the page.
Currently you have no way of getting to the returned letter page. This
commit adds a link to it from the dashboard, following the pattern of
the new received text messages banner.
We found another scenario where signing out of the db can cause a 500.
If the user archives their trial mode service, current_service.active = false, then signs out, the current user was being signed out client side first, meaning current_user is now an Anonymous user, next the call to the API is made to log out user on db, all calls to NotifyApiClient `check_inactive_service`, which is only authorised if user is platform_admin, but an AnonymousUser does not have that attribute, so a 500 is raise.
Seemed a bit cleaner to change the User.signout method to rather than the `check_inactive_service` method for current_user.is_authenticated.
Anytime a user clicks "sign out" we should be signing them out server side as well. This can be accomplished by setting the Users.current_session_id = null.
I found that the method User.logged_in_elsewhere doesn't need to check if the current_session_id is None. The current_session_ids in the cookie and db (redis or postgres) then the user should be forced to log in again.
service contact blocks contain new lines - and jinja2 normally ignores
newlines (as in it keeps them as new lines) - but we need to turn them
into `<br>` tags so that we can show the formatting that the user has
added. We were previously just doing `{{ block | nl2br | safe }}`. nl2br
turns the new lines into `<br>` tags, and then `safe` tells jinja that
it doesn't need to escape the html.
this causes issues if the user adds `<script>alert(1)</script>` to their
contact block (or some other evil xss hack), where that will get let
through due to the safe flag
To solve this, use `Markup(html='escape')` to sanitise any html, and
then convert new lines to <br>.
bump utils
another xss
