When we combine simplified polygons from different areas we try to use
the polygons that are defined in metres as a speed optimisation.
This doesn’t work when those polygons are in several different
projections because the distance in metres is relative to the edge of
the projection. Just naively choosing one projection means that some
of the polygons will shift position on the earth by 100s of kilometers
once they are converted back to degrees.
To avoid this we need to:
- check for situations where we have polygons in multiple different
projections
- transforms these polygons back into degrees and let `Polygons` pick a
common projection for all of them before doing any
merging/smoothing/etc.
Looking up the coordinate reference system for a given polygon’s bounds
is surprisingly expensive.
We can make things run faster by precomputing it at the time of
generating the simplified polygons, then storing it in the SQLite
database alongside the polygon data.
Transforming full resolution polygons to linear coordinates is somewhat
expensive. We can make things run faster by:
- looking at `simple_polygons` which have fewer points and are therefore
faster to transform.
- reusing polygons that have already been transformed where possible
Brings in https://github.com/alphagov/notifications-utils/pull/889/files
At the moment, we are not doing any transformation of features before
applying geometric algorithms to them. This is, in effect, assuming that
the earth is flat.
This new version of utils implements the transformation of our polygons
to a Cartesian plane. In other words, it converts them from being
defined in spherical degrees to metres.
For the admin app this means we need to convert places where the code
expects things to be measured in degrees to work in metres instead.
`reference` isn’t very human-friendly – the Environment Agency just
supply a UUID in this field.
The Environment Agency also populate the `<event>`` field with some
human readable text, for example:
> 013 Issue Severe Flood Warning EA
(013 is an ‘area code’ which will be meaningful to the Flood Warning Service team)
This commit changes the frontend to display the value of the `cap_event`
field, if it’s present, which is where the API stores the value of the
`<event>` field from the original CAP XML.
***
Depends on:
- [x] https://github.com/alphagov/notifications-api/pull/3344/files
The original code to raise the exception was flawed: if a broadcast
only had a single area that was invalid, we would assume it was a
custom broadcast [1]. Since the recent changes [2] fixed the flaw
we're now getting exceptions for broadcasts of this kind.
It's not practical to go and manually fix the invalid broadcasts,
and the likelihood is there will be more in future as the set of
areas we support changes. This takes a pragmatic approach of simply
logging the issue and pretending such broadcasts are custom.
[1]: 926ada2f21
[2]: https://github.com/alphagov/notifications-admin/pull/4014/files#diff-2dd8f77d6df281e7674b20263cdf27a3d58b839dc5930c0087ac8b9749b313e4R92
Previously we relied on the API defaulting this field to an empty
array [1], but that conflicts with using it to decide whether a
broadcast is custom or created in this app.
[1]: 3779146cc5/app/models.py (L2342)
Custom broadcasts created directly via the API app won't have area
IDs since [1], where we started to distinguish between "names" (all
broadcasts have these) and IDs (for broadcasts created in this app).
We forgot to propagate the distinction into the code here.
This code fixes the bug for all broadcasts created after [1]. Any
custom broadcasts created before [1] will have their "ids" field set
instead of "names" so we won't show anything for them. This seems
reasonable as we don't support custom broadcasts yet.
[1]: 023a06d5fb
The aggregate names don't need to be sorted, but it helps to do
this for the tests, since the implementation of sets may not be
stable between machines and lead to sporadic test failures.
I did also toy with sorting granular area names so they have a
similar ordering, but this would take them out-of-order with IDs
and really the important thing is that the ordering is stable.
Depends on: https://github.com/alphagov/notifications-api/pull/3314
Previously:
- We introduced a new "areas_2" API field that's can
populate the "areas" column in the same way.
- We updated the Admin app to use the new field, so that
the old "areas" and "simple_polygons" API fields are unused.
- We repurposed the unused "areas" API field to use
the new "areas_2" format.
This PR:
- We can switch the Admin app back to the "areas" field,
but in the new format.
Future PRs:
- Remove support for the unused "areas_2" field (migration complete)
This will be run with a CSV of all broadcast messages. Since very
few users are creating or updating broadcasts, it's highly unlikely
we'll encounter a race condition during the update.
These will be used as a fallback for display in gov.uk/alerts. It
also helps to have them in the DB to aid in quickly identifying
where an alert was sent, which is hard from the IDs.
We will look at backfilling names for past alerts in future work.
Previously we just held the new area_ids in memory. Setting them
on the object means we can reuse its functionality to get polygons
and also avoids confusion if in future we try to continue using
the object after calling "add_areas" or "remove_areas".
Theoretically the maximum expiry time of a broadcast should be 24 hours.
If it goes over 24 hours there can be problems.
However we want to make it more conservative to mitigate two potential
issues:
1. The CBC has a repetition period (eg 60 seconds) and a count (eg
1,440). If these were slightly innaccurate or generous it could take
us over 24 hours. For this reason we should give ourselves half an
hour of buffer.
2. It’s possibly that the CBC could interpret a UTC time as BST or vice
versa. Until we’re sure that it’s using UTC everywhere, we need to
remove another whole hour as buffer.
In total this means we remove 1 hour 30 minutes from 24 hours, giving an
expiry time of 22 hours 30 minutes.
While testing alerts on these channels the MNOs sometimes need to
restart their CBCs to make sure everything is failing over properly.
If the CBC does not come back up, for whatever reason, then we are left
in a state where the alert can’t be cancelled.
To minimise the impact to the public in this scenario we should keep the
expiry time at 4 hours for alerts sent on test channels. We recently
increased it back up to 24 hours for all channels, so this in effect is
reverting that change for channels that won’t be used in a real
emergency.
The page which shows the count of phones does some logic based on how
close the ‘will get’ and ‘likely to get’ numbers are. This means it
accesses the `BroadcastMessage.count_of_phones` and
`BroadcastMessage.count_of_phones_likely` properties multiple times.
These properties are computed fresh every time, and are quite expensive
to compute. By caching them in memory we can cut the page load time
approximately in half.
Our current assumption is that the bleed area has the same population
density as the broadcast area.
This is particularly naïve when:
- the bleed area overlaps the sea – no-one lives in the sea
- the broadcast area is a village and the bleed area is the surrounding
countryside
- the broadcast area is adjacent to a densely populated area like a city
We can be smarter about this now that we have a way of determining the
number of phones in an arbitrary area, based on the known areas that we
have population data about.
Calculating the population in an overlap is a slightly more intensive
calculation. So we only doing it for areas which are smaller enough that
it doesn’t slow things down too much. For larger areas we still use the
more naïve algorithm.
Previously this was hidden away in an anonymous __init__.py file.
I did think about splitting the models into individual files, like
we do with the top-level models for the app. Since the models are
only imported in one place - i.e. are all used together - it didn't
seem worth the hassle, so I've kept them in one file.
The code for this page was making assumptions about properties which
aren’t present on rejected broadcasts.
This commit accounts for those properties and presents the relevant
elements on the page.
There are basically two kinds of 4G masts:
Frequency | Range | Bandwidth
----------|-------------|----------------------------------
800MHz | Long (500m) | Low (can handle a bit of traffic)
1800Mhz | Short (5km) | High (can handle lots of traffic)
The 1800Mhz masts are better in terms of how much traffic they can
handle and how fast a connection they provide. But because they have
quite short range, it’s only economical to install them in very built up
areas†.
In more rural areas the 800MHz masts are better because they cover a
wider area, and have enough bandwidth for the lower population density.
The net effect of this is that cell broadcasts in rural areas are likely
to bleed further, because the masts they are being broadcast from are
less precise.
We can use population density as a proxy for how likely it is to be
covered by 1800Mhz masts, and therefore how much bleed we should expect.
So this commit varies the amount of bleed shown based on the population
density.
I came up with the formula based on 3 fixed points:
- The most remote areas (for example the Scottish Highlands) should have
the highest average bleed, estimated at 5km
- An town, like Crewe, should have about the same bleed as we were
estimating before (1.5km) – Pete D thinks this is about right based on
his knowledge of the area around his office in Crewe
- The most built up areas, like London boroughs, could have as little as
500m of bleed
Based on these three figures I came up with the following formula, which
roughly gives the right bleed distance (`b`) for each of their population
densities (`d`):
```
b = 5900 - (log10(d) × 1_250)
```
Plotted on a curve it looks like this:
This is based on averages – remember that the UI shows where is _likely_
to receive the alert, based on bleed, not where it’s _possible_ to
receive the alert.
Here’s what it looks like on the map:
---
†There are some additional subtleties which make this not strictly true:
- The 800Mhz masts are also used in built up areas to fill in the gaps
between the areas covered by the 1800Mhz masts
- Switching between masts is inefficient, so if you’re moving fast
through a built up area (for example on a train) your phone will only
use the 800MHz masts so that you have to handoff from one mast to
another less often
This adds to the `__le__` method on the `BroadcastMessage` class to
allow two BroadcastMessages with no `updated_at` to be compared.
Previously, the method expected at least one message to have
`updated_at` set, but this is not necessarily the case.
Broadcasts created by the API are different in that:
- they aren’t created by any user, so don’t have a `created_by_id`
- they are created instantly, not in steps, so don’t have an
`updated_at` time
This commit alters the views to account for when these pieces of
information aren’t present.
At the moment the admin app expects all broadcasts to have a template,
and expects the content of the alert to come from the template.
This commit makes it so those pages can still get a `Template` instance,
but populated with content straight from the `content` field in the
database.
We think that in some cases alerts will be composed in the moment, and
therefore making people first create a template is:
- not a good use of their time
- adding some conceptual complexity which they don’t need
This commit makes it possible to type some words and have them go
straight into the `content` field in the database.
In the future we might want to progressively enhance the radio buttons
so they show on the same page (like we do with the grey buttons on the
templates page).
We have lots of functions for converting various types of data into
strings to be displayed to the user somewhere.
This commit collects all these functions into their own module, rather
than having them cluttering up `app/__init__.py` or buried amongst
various other things that have ended up in `app/utils.py`.
For emails and text messages we sort by the time the user (or API) sent
them.
This makes sense for broadcasts too, since most users will receive the
alert within seconds of it being broadcast.
For alerts that haven’t started yet we can sort by `updated_at`, which
is when the user preparing the broadcast submitted it for approval.
Our style for areas is pale blue background with black keylines or bold
black text.
This commit makes the display of area names on the dashboard consistent
with that visual style.
This also means that we’re not truncating the list of areas, which is
appropriate because no one area is more important than any of the
others.
We’ve had some feedback from user research that difference between
‘will get alert’ and ‘likely to get alert’ is not clear, and it’s hard
to tell if the latter is inclusive of the former. This leads people to
question the validity of these numbers, which is important, because an
the estimate should give you some idea of the impact of what you’re
about to do.
This commit reformats the number as a range, for example 1,000 to 2,000
phones.
If the range is small, eg 40,000,000 to 40,800,000 then this suggests
a false level of accuracy. So instead we just give one number and say
it’s an estimate, eg ‘40,000,000 phones estimated’
If you’re adding another area to your broadcast it’s likely to be close
to one of the areas you’ve already added.
But we make you start by choosing a library, then you have to find the
local authority again from the long list. This is clunky, and it
interrupts the task the user is trying to complete.
We thought about redirecting you somewhere deep into the hierarchy,
perhaps by sending you to either:
- the parent of the last area you’d chosen
- the common ancestor of all the areas you’d chosen
This approach would however mean you’d need a way to navigate back up
the hierarchy if we’d dropped you in the wrong place. And we don’t have
a pattern for that at the moment.
So instead this commit adds some ‘shortcuts’ to the chose library page,
giving you a choice of all the parents of the areas you’ve currently
selected. In most cases this will be one (unitary authority) or two
(county and district) choices, but it will scale to adding areas from
multiple different authorities.
It does mean an extra click compared to the redirect approach, but this
is still fewer, easier clicks compared to now.
This meant a couple of under-the-hood changes:
- making `BroadcastArea`s hashable so it’s possible to do
`set([BroadcastArea(…), BroadcastArea(…), BroadcastArea(…)])`
- making `BroadcastArea`s aware of which library they live in, so we can
link to the correct _Choose area_ page
At the moment there are some areas which have:
- a `count_of_phones` value of `None`
- no sub-areas
This is wrong, but until we fix the data the phone counting code needs
to handle this.
This commit:
- adds the `or 0` in the right place (where it will catch these areas
with missing data)
- adds a test which checks these areas, and compares them to other kinds
of areas
We need to give people a better feel for the consequences of
broadcasting an alert. We’ve seen in research that some users will
assume it is subscription based, or opt-in, rather than going to every
phone in the area.
I reckon that the most effective way to communicate this is to put some
numbers next to the areas, to give people an idea of how many people
will get alerted.
We can estimate how many phones are in an area by:
- taking the population of all electoral wards in that area
- multiplying it by the percentage of people who own an internet
connected phone[1]
The Office for National Statistics publish both these datasets.
The number of people who own an intenet connected phone varies a lot by
age. Since the population data for each ward is broken down by age we
can factor this in. Simplified, the calculation looks like this:
- take the _Abbey_ ward of _Barking and Dagenham_
- in this ward there are 26 people aged 80
- 40% of people over 65 have an internet-connected phone
- therefore 10 of these 80-year-olds would be likely to receive a
broadcast
- (repeat for all other ages)
These numbers won’t be exact, but should be enough to give people a feel
for the severity of what they’re about to do. We can see if they acheive
this aim in user research.
1. This is a proxy for the number of people who are likely to have a 4G
capable phone, because only 4G capable phones will be receiving
broadcasts to begin with
When creating broadcast message, or updating it.
We want to send simple polygons to API so we can
later relay them to the broadcast provider.
Test that update broadcast message sends polygons correctly
We have a bunch of stuff for doing lat/long transformation in the
`BroadcastMessage` class. This is not a good separation of concerns, now
that we have a separate class for dealing with polygons and coordinates.
