On the uploads page we only show jobs which are within a service’s data
retention.
This commit does the same for when we’re listing the jobs for a contact
list. This matches the UI, which says a contact list has been ‘used
`<count_of_jobs>` in the last <data_retention> days’
It’s a bit unintuitive that starting a job from a contact list makes a
copy of the file, which has no relationship to the list it was copied
from. This is more of an implementation detail, rather than something
that comes from people’s mental models of what is going on. Or at least
that’s what I hypothesise.
I think it’s clearer to show jobs that come from contact lists within
the lists that they were created from. By naming the jobs by template
this gives a clearer view of what messages have been sent to the group
over time.
Now that scheduled jobs are mixed in with regular jobs it looks weird
for the sort order to be different. This makes the sort order
consistently go from furthest in the future to furthest in the past.
The old sort order made sense when scheduled jobs were displayed
separately on the dashboard.
The API clients should just deal with calling the API and returning the
data from it.
Inferring things from the data is more logically done at the model
layer. But we couldn’t do that before, because we didn’t have a model
layer for jobs.
New units tests have not been written for this page because it is very like this will be refactor and probably a new template created for the page. Some design needs to go into this page.
But we needed something ready for user research.
when clients are defined in app/__init__.py, it increases the chance of
cyclical imports. By moving module level client singletons out to a
separate extensions file, we stop cyclical imports, but keep the same
code flow - the clients are still initialised in `create_app` in
`__init__.py`.
The redis client in particular is no longer separate - previously redis
was set up on the `NotifyAdminAPIClient` base class, but now there's one
singleton in `app.extensions`. This was done so that we can access redis
from outside of the existing clients.
We have some teams who haver a series of files they have to send each
day. It’s easy to get muddled up and accidentally send the same file
again, if you think you haven’t already sent it.
This commit blocks you from sending the same combination of template
version and filename more than once on the same day[1].
This won’t affect teams who re-use the same template to give (for
example) updates on an incident for business continuity. These teams
edit the template between each send, thereby updating the version
number of the template.
1. This is based on how the `limit_days` argument to the API works - you
can dig into the code here: 2bd4f74ad0/app/dao/jobs_dao.py (L50)
This removes some code which is duplicative and obscure (ie it’s not
very clear why we do `"a" * 73` even though there is a Very Good Reason
for doing so).
In the long term, we don't want to show cancelled letters. But for now,
this changes cancelled letters to display in the same way that letters
with a status of permanent-failure, since we are currently giving
letters that we want to cancel the status of permanent failure.
Calling `.set()` with `True` stores the byte string `'True'` which
cannot subsequently be decoded from JSON (because boolean values in
JSON are lowercase, ie `true`).
> "normal" is a word that does not really tell much, could we maybe name
> this constant better?
‘Non-scheduled’ more explicitly describes that statues in this set.
At the moment the dashboard does two API calls to find out if a service
has:
1. Scheduled jobs
2. Normal jobs
API calls are slow because they are synchronous, go over the network and
touch the database. We can’t cache these API calls because:
- a scheduled job could become a normal job at any time
- the statistics on a normal job are constantly updating
However there are plenty of services which don’t have any jobs, and
probably never will. And finding out if a service has any jobs is
reliably cacheable (because as soon as a service creates its first job
it has some jobs).
So this commit:
- refactors the way we get scheduled/normal jobs into the job_api_client
to make the view a bit slimmer
- makes an additional, Redis-wrapped call to find out if any jobs exist
before trying to get the jobs
This should result in a speedup on the dashboard, and can be used in the
future if there’s anywhere else we want to show or hide something
depending on whether a service has created any jobs (I have some ideas).
Currently requests to the API made from the admin app are going from
PaaS admin app to the nginx router ELB, which then routes them back
to the api app on PaaS.
This makes sense for external requests, but for requests made from
the admin app we could skip nginx and go directly to the api PaaS
host, which should reduce load on the nginx instances and
potentially reduce latency of the api requests.
API apps on PaaS are checking the X-Custom-Forwarder header (which
is set by nginx on proxy_pass requests) to only allow requests going
through the proxy.
This adds the custom header to the API client requests, so that they
can pass that header check without going through nginx.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
Because we’re setting the API key and service ID after calling the
`__init__` method of the client it wasn’t doing the thing where it
splits the combined key into the two individual UUIDs. So we still need
to set them directly, individually on the client.
The Notify API client changed in version 4 to take two arguments, not
three (service ID was removed in favour of the combined API key).
This gets a bit gnarly because the API key has to be at least a certain
length so it can be substringed internally.
The clients never get passed useful values to their `__init__` methods.
Rather the real values are passed through later using the `init_app`
method.
So it should be an error if the client is relying on the values that
get passed to it’s init method. Easiest way to ensure this is by making
the `__init__` method not expect any arguments and passing fake values
to the `Super` call.
If you schedule a job you might change your mind or circumstances might
change. So you need to be able to cancel it. This commit adds a button
on the job page which hits the `…/cancel` API endpoint for a job.
The create job endpoint returns the data about the job with a `data:`
wrapper. This commit makes sure that, when the client is trying to
process a job which has just been created, it looks inside the `data`
wrapper.
Users need to pick a time in the next 24hrs, or send a file immediately.
Rationale for this is a bit lost in time-before-holiday, but generally:
‘Now’ and ‘later’ as the inital choices makes it really clear what
this feature is about conceptually.
The choice of times is absolute, eg ‘1pm’ not ‘in 3 hours’
This is just some refactoring.
`defaultdict` is a data structure which won’t raise a `KeyError` if you
try to access a key that doesn’t exist.
By passing `int` as the first argument, trying to access the value of
any key that doesn’t exists will return the value of `int()`, ie `0`
It’s useful to know how many notifications we’ve handed off to our
providers. This is a measure of how complete the processing of the job
is.
This is important, because once the job processing is complete then you
can accurately reconcile the report with the CSV file that you’ve
uploaded.
