Converting Python data to CSV makes every field a string. This means
that in the report we return to the user every field will be a string,
even if it’s come from an `int` type in Python. This is because the CSV
‘standard’ doesn’t support any kind of typing.
Excel does support types for fields, so we can make our reports more
useful by preserving these types. This is particularly relevant in the
report we generate for performance platform, which needs the `count`
column to be a number type.
This commit adds extra code paths to the `Spreadsheet` class which mean
that it can be instantiated from either CSV data or a list of Python
data. Previously we were converting the Python data to CSV as an
intermediate step, before instantiating the class.
Separated s3_client.py into 3 files - for logos, CSV files and the MOU.
This helps to keep things clearer now that we need to add lots more logo
functions for letters.
This commit adds content pages for the notifications pages, particularly
the letter pages, which will make things clearer now that we will soon be allowing
letters to be cancelled.
The main changes are:
* The confirmation banner for letters sent from a CSV file now states when
printing will start.
* We state the CSV file that notifications were sent from on the
notifications page
* The notification page for letters shows when printing starts (today,
tomorrow, or that date that the letter was printed)
This is useful if you have lots of people sending messages and want to
report on who’s doing what.
Needs the API updating to return `created_by_name` in its response.
Because we alias domains (eg `foo.gsi.gov.uk` to `foo.gov.uk`, or where
a local council has multiple domains) it could be hard to look up a
brand (which has one domain field).
Therefore we need a way of getting the canonical domain from a user’s
email address, which we can later use to look up their branding.
we were seeing isort produce different outputs locally and in docker -
this was due to it having different opinions about whether the tests
module (ie all our unit tests) is a first party (local) or third party
(pip installed) import. It's a first party import, so by defining this
in the setup.cfg isort settings, we can force it to be consistent
between environments.
Note: I don't know why it was different in the first place though
Previously, we were looking at the day of the week - so messages sent
six days ago would show up as "tomorrow". We now look at the actual
date, so that won't happen again.
We were also subtracting an hour to make 00:00 this evening show up as
"midnight today", despite it technically being tomorrow. However, this
means that 00:59 tomorrow morning would show up as "00:59 today", a
full day out. So reduce that to just a minute, so it doesn't affect
other times of day.
This makes it easier to write a good message in the request to go live
submission. And encapsulating it in the `GovernmentDomain` class keeps
the view nice and clean.
If a cell in the original file contains a comma, it comes back as two
cells in the downloaded file.
The CSV writer has logic to deal with this. It seems to work a lot
better that just concatenating the columns with commas ourselves.
When downloading a report of a which messages from a job have been
delivered and which have failed we currently only include the Notify
data. This makes it hard to reconcile or do analysis on these reports,
because often the thing that people want to reconcile on is in the data
they’ve uploaded (eg a reference number).
Here’s an example of a user talking about this problem:
> It would also be helpful if the format of the delivery and failure
> reports could include the fields from the recipient's file. While I
> can, of course, cross-reference one report with the other it would be
> easier if I did not have to. We send emails to individuals within
> organisations and it is not always easy to establish the organisation
> from a recipient's email address. This is particularly important when
> emails fail to be delivered as we need to contact the organisation to
> establish a new contact.
– ticket 677
We’ve also seen it when doing research with a local council.
This commit takes the original file, the data from the API, and munges
them together.
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
There’s over 1000 domains in our file. This is too much for parametrize
to handle when running the tests on multiple cores. End up with this
error:
```
Different tests were collected between gw1 and gw2.
```
The thing that matters for which agreement an organisation has to sign
is whether or not that organisation is crown or non-crown.
There is only a partial overlap between crown/non-crown and
local/central. We can’t infer one fro the other. So this commit makes it
explicit by marking all local government organisations as non-crown,
which is something we can know for sure.
We don’t, for example, know the inverse, that all parts of all central
government organisations are crown bodies (but we can mark some of them
as being so later on).
This adds information about which orgs have signed an MOU to the domain
list. The meaning of the attribute is:
- `true`: MOU signed for the whole organisation
- `false`: no MOU for any part of the organisation
- `null` (or missing): can’t be sure if it’s true or false
This commit:
- makes the logic around looking up a domain a bit more sophisticated
by matching on the longest domain name first
- exposes the details about an organisation to consumers of the
`GovernmentDomain` class
It will also be useful to know (especially for the API):
- when a letter was printed
- if it’s been printed or not
This commit:
- adds code to calculate these two pieces of information
- refactors the function to return a `namedtuple` – a tuple of two items
was manageable, but with four items it was getting hard to know what
each one meant – this lets us label each piece of information that is
being returned
Letter delivery depends on:
- how long it takes to print
- how long it takes to post
Both of these process are impacted by weekends, because people don’t
work on weekends.
It also depends on if you submit your letter before or after 5pm,
because that’s the cut off time for getting a letter printed on a given
day – ie after 5pm on Monday is effectively the same as Tuesday and so
on.
But I reckon all our users need to know is roughly how long it will take
until the letter turns up on the user’s doorstep. So this commit adds
a function to calculate this. Doesn’t surface it on the front end _yet_.
`test_utils.py` is actually a file for testing some utils, not
some utils for testing.
Test helper functions and fixtures are better placed in conftest,
and also means that we can use them in conftest.
This commit adds two things:
a section on the dashboard to show how many inbound messages the
service has received in the last 7 days, and how recently an inbound
message has been received
---
Doesn’t show the contents of any messages, just like how the rest of the
dashboard is an aggregation, never individual messages.
a page to show all the inbound messages the service has received in
the last 7 days
---
This shows the first line of the message. Eventually this will link
through to a ‘conversation’ page, where a service can see all the
messages it’s received from a given phone number.
The send yourself a test feature is useful for two things:
- constructing an email/text message/letter without uploading a CSV file
- seeing what the thing your going to send will look like (either by
getting it in your inbox or downloading the PDF)
- learning the concept of placeholders, ie understanding they’re thing
that gets populated with _stuff_
The problem we’re seeing is that the current UI breaks when a template
has a lot of placeholders. This is especially apparent with letter
templates, which have a minimum of 7 placeholders by virtue of the
address.
The idea behind having the form fields side-by-side was to help people
understand the relationship between their spreadsheet columns and the
placeholders. But this means that the page was doing a lot of work,
trying to teach:
- replacement of placeholders
- link between placeholders and spreadsheet columns
The latter is better explained by the example spreadsheet shown on the
upload page. So it can safely be removed from the send yourself a test
page – in other words the fields don’t need to be shown side by side.
Showing them one-at-a-time works well because:
- it’s really obvious, even on first use, what the page is asking you to
do
- as your step through each placeholder, you see the message build up
with the data you’ve entered – you’re learning how replacement of
placeholders works by repetition
This also means adding a matching endpoint for viewing each step of
making the test letter as a PDF/PNG because we can’t reuse the view of
the template without any placeholders filled any more.
before each request, we put the current service on the flask session,
except for with the static folder, cos it's not needed.... except, if
we 404, then we return the 404 template, which checks if you're logged
in or not to display different nav bar items. This was crashing when
current_service wasn't set, so we now set it.
also cleaned up some imports and stuff in test files
* Remove formatting (most performance intensive) as this will happen on API
* Remove buffering of notifications so we can yield them back straight away
* Refactor existing tests to match new format of the notification resp for csv
Right now we tell people that the usage page is for the current
financial year. This is a lie – it’s for all time.
So this commit calls through to the API to get the stats for (by
default) the current financial year.
We already do this for the monthly breakdown, this just does the same
thing for the yearly totals.
It also adds navigation to show the data for other financial years:
- previous so you can go back and see your usage and verify that the
bill you’re about to pay is correct
- next so that you can check what your SMS allowance is going to be
before you actually get into it
