This did more than it said and had some unconventional behaviour:
- It modified the input data. We can avoid this by computing the
postage group on-the-fly and using "sorted" instead of "sort".
- It defined a custom, named tuple. This isn't necessary as Jinja
allows us to access elements by qualification (".") already.
We can also use the same lambda function to group and sort items,
since the sort predicate is the same one we use to group them.
This is now only used for letters and represents the number sent
[^1]. We could use the chargeable_units field, but using "_sent"
is more consistent with the annual attributes [^2].
In fact, chargeable_units isn't actually used anywhere, but I've
kept it in the test data as it is part of the real API and helps
clarify the other values for SMS - free vs. charged.
Note: for SMS I've used an arbitrary "1234" for "chargeable_units"
to indicate it's not used and may be different to the number sent -
for SMS it's related to the number of fragments.
[^1]: bb62d22f25/app/dao/fact_billing_dao.py (L339)
[^2]: 3a1ac189ff/app/main/views/dashboard.py (L339)
This starts using the sms_{cost, charged, free_allowance_used}
fields in the new API to replace the "get_free_paid_breakdown"
function we had before, which could not support multiple rates.
In order to use "get_free_paid_breakdown" the calling method had
to store a "cumulative" variable to calculate the free allowance
used so far, which is now done by the API.
To calculate the data for conftest.py, I had to start from the
bottom ("April") and manually calculate the free allowance used
to emulate the API - this is what "cumulative" used to do.
This is also an opportunity to DRY-up the filtering of usage by
month, which we will reuse in the following commits. Doing a sum
is simple enough that it can be done inline, avoiding indirection.
Note: I've removed the pricing assertion in the "0_free_allowance"
test as it's covered elsewhere - the value of the test is really to
check that we don't show the remainder if there never was any.
The previous, manual calculation could be incorrect depending on
which SMS rates the free allowance was attributed to.
The new field also supersedes the old "letter_total" bolt-on so we
can get cost information consistently for both types.
This will make the following changes clearer.
In the next commits we'll go into more detail about "billing_units"
and how it differs for SMS vs. emails and letters.
At the moment, we put the sms rate on the usage page for each
months billing data by taking the single sms rate for the year.
The assumption that there will be a single sms rate for the year is
no longer going to be true. Therefore, instead we take the sms
rate from the monthly data itself which tells us the rate for
a months worth of sent SMS.
Some users have reported a problem with the received text message
report:
> I have tested the reply service but in the excel report the mobile
> number is showing as 4.47900E+23. How can I change the format so that
> it is show the mobile number that has replied?
This is happening because Excel is interpreting a phone number in the
format `447900900123` as a number in
[scientific notation](https://en.wikipedia.org/wiki/Scientific_notation),
in other words 4.479 × 10<sup>23</sup>.
`447900900123` is the format that our provider is giving us the number
in – there’s no guarantee it will always be in this format.
We can prevent this behaviour by putting spaces in the numbers. Excel
and Google Sheets won’t try to convert a string with spaces into a
number.
I think we used to do this for the sent text messages report but
probably stopped because we decided it was better to keep the phone
number in the same format as it had been supplied to us for
reconcilliation purposes.
Previously this was duplicated between the "two_factor" and the
"webauthn" views, and required more test setup. This DRYs up the
check and tests it once, using mocks to simplify the view tests.
As part of DRYing up the check into a util module, I've also moved
the "is_less_than_days_ago" function it uses.
now that we no longer set it since
https://github.com/alphagov/notifications-admin/pull/3841 was merged, we
don't need to remove it either. And we can remove checks that expect it
when cleaning up the session. And the unit tests that make sure we
ignore it if it's in the session.
So long, session['invited_user'] and session['invited_org_user']!
We have lots of functions for converting various types of data into
strings to be displayed to the user somewhere.
This commit collects all these functions into their own module, rather
than having them cluttering up `app/__init__.py` or buried amongst
various other things that have ended up in `app/utils.py`.
The api returns letter details split by postage, so international
letters are returned with a postage of `europe` or `rest-of-world` not
`international` and these rows need to be added together when the rate
is the same before they are displayed on the usage page.
To do this, we need to replace the postage of `europe` and
`rest-of-world` with `international`. The data then needs to be sorted
by postage and rate before the letter units for rows which are
international and have the same rate are added together.
No functional changes, but this changes the letter details that are
used for the usage page from a tuple to a named tuple since this makes
it easier to understand.
There are lots of places where we redirect people to the dashboard, for
example after logging in. Rather than add logic to all these places to
check for the broadcast permission, let’s just redirect from the normal
dashboard to the broadcast dashboard.
Other places like the main navigation can still link directly to the
broadcast dashboard, where we do care about speed and not going through
multiple redirect jumps.
From a question on cross-government Slack:
> re the Usage tab - currently it shows 3 financial years - last year,
> this year and next year. is it possible to replace the "next year" tab
> with something more useful? its always going to be blank! I was
> thinking it would be good to have 2 financial years ago, 1 financial
> year ago and this financial year.
This seems like a reasonable idea, and is something we’ve talked about
before. The original intention[1] was that seeing your (unchanged) free
allowance for next year would be useful, but that doesn’t really seem to
be a user need.
***
1. See https://github.com/alphagov/notifications-admin/pull/1094
> so that you can check what your SMS allowance is going to be before
> you actually get into it
We’ve done this already for services with the upload letters permission.
And all services can upload letters now.
But we’re still returning it in the JSON response we use to AJAX-ify the
page.
Since the jobs response can query stats for up to 50 jobs at a time this
puts some load on the API/database. Hopefully this might drop that load
a bit.
As per https://www.pivotaltracker.com/story/show/170796514 we want to make the delete template confirmation dialog box more consistent and clear.
The API has been updated with a new endpoint that only returns the last-used date, this date is more accurate since it goes to the ft_notification_status table, if the notification table is empty.
This follows the pattern of what we’ve done with services, users and
events.
It gives us a way of neatly instantiating a model for each item in the
list we get back from the API and reduces the complexity of the view
layer code.
Now is a good time to do this because we’re going to be making a bunch
of changes to the jobs pages, and those changes will be easier to code
and understand with a sensible model behind them.
We mostly rely on the API returning a 404 to generate 404s for trying
to get things with non-UUID IDs. This is fine, except our tests often
mock these API calls. So it could look like everything is working fine,
except the thing your passing in might never be a valid UUID, and thus
would 404 in a non-test environment.
So this commit:
1. uses the `uuid` URL converter everywhere there’s something that looks
like an ID in a URL parameter
2. adds a test which automates checking for 1.
This reverts 1b1839ad30, which removed
the usage from the dashboard because it was causing performance
problems:
> **The yearly usage section on the dashboard page takes too log as a
> result services with large yearly stats are timing out.**
>
> As a short term fix we have taken the yearly stats off the dashboard.
>
> There is a plan to create permanent statistic tables to warehouse the
> data.
The long term fix (the fact tables) is now in place, so it should be OK
to bring this back.
This is part of a wider piece of work to refresh the dashboard page now
that jobs are moving to their own page.
We hid letters originally because it wasn’t a mature feature. We rolled
it out by letting teams choose to use it (#1803)
and then automatically giving it to new teams (notifications-api/#1600).
This commit doesn’t change who has access to letters, but it does make
it more discoverable by revealing it in the UI. This is the same thing we do for emails/texts, where even if you switch them off they still show up on the dashboard and usage
page.
Even if your service doesn’t send letters now, it might have done
previously.
The original reason for hiding letters was because it wasn’t a mature
feature. But now that it is, we should make it discoverable even for
existing teams. So that means not conditionally hiding it.
This is the same thing we do for emails/texts, where even if you switch
them off they still show up on the dashboard and usage page.
