The Python rtree library we are using to build RTrees has a dependency
on the C package libspatialindex. This package is not installed on PaaS,
so it’s hard for us to use it.
This commit changes the code to use a library called rtreelib instead.
rtreelib doesn’t have a built in way to serialise the index it builds,
so I’ve had to implement that using pickle.
We want to know how many phones are in a user-supplied polygon, so we
can show the impact of a broadcast, in the same way that we do when
users pick areas from our library.
We already know how many phones are in each electoral ward. But there
are challenges with an arbitrary polygon:
- where it does overlap a ward, the overlap could be partial
- it could overlap more than one ward
- finding out which wards it overlaps by brute force (looping through
all the wards and seeing which ones intersect with our polygon) would
be way to slow to do in real time
Instead we can use a data structure called an R-tree[1] to build an
index which provides a much, much faster way of looking up which
polygons overlap another. We can build this tree in advance and save it
somewhere, which means there’s a lot of computation we don’t need to do
in real time.
The R-tree returns a set of objects (ward IDs) which we can go and look
up in our library of electoral wards. These wards will be the ones that
might have some overlap with our custom polygon.
Once we have this small set of wards which might overlap our ward, we
can look at the size of the area of overlap (relative to the size of the
whole ward) and multiply that by the known count of phones in that ward
to get an approximation of the count of phones in the overlap area.
Summing these approximations give an estimate for the whole area of the
custom polygon.
1. https://en.wikipedia.org/wiki/R-tree
If you’re adding another area to your broadcast it’s likely to be close
to one of the areas you’ve already added.
But we make you start by choosing a library, then you have to find the
local authority again from the long list. This is clunky, and it
interrupts the task the user is trying to complete.
We thought about redirecting you somewhere deep into the hierarchy,
perhaps by sending you to either:
- the parent of the last area you’d chosen
- the common ancestor of all the areas you’d chosen
This approach would however mean you’d need a way to navigate back up
the hierarchy if we’d dropped you in the wrong place. And we don’t have
a pattern for that at the moment.
So instead this commit adds some ‘shortcuts’ to the chose library page,
giving you a choice of all the parents of the areas you’ve currently
selected. In most cases this will be one (unitary authority) or two
(county and district) choices, but it will scale to adding areas from
multiple different authorities.
It does mean an extra click compared to the redirect approach, but this
is still fewer, easier clicks compared to now.
This meant a couple of under-the-hood changes:
- making `BroadcastArea`s hashable so it’s possible to do
`set([BroadcastArea(…), BroadcastArea(…), BroadcastArea(…)])`
- making `BroadcastArea`s aware of which library they live in, so we can
link to the correct _Choose area_ page
We need to give people a better feel for the consequences of
broadcasting an alert. We’ve seen in research that some users will
assume it is subscription based, or opt-in, rather than going to every
phone in the area.
I reckon that the most effective way to communicate this is to put some
numbers next to the areas, to give people an idea of how many people
will get alerted.
We can estimate how many phones are in an area by:
- taking the population of all electoral wards in that area
- multiplying it by the percentage of people who own an internet
connected phone[1]
The Office for National Statistics publish both these datasets.
The number of people who own an intenet connected phone varies a lot by
age. Since the population data for each ward is broken down by age we
can factor this in. Simplified, the calculation looks like this:
- take the _Abbey_ ward of _Barking and Dagenham_
- in this ward there are 26 people aged 80
- 40% of people over 65 have an internet-connected phone
- therefore 10 of these 80-year-olds would be likely to receive a
broadcast
- (repeat for all other ages)
These numbers won’t be exact, but should be enough to give people a feel
for the severity of what they’re about to do. We can see if they acheive
this aim in user research.
1. This is a proxy for the number of people who are likely to have a 4G
capable phone, because only 4G capable phones will be receiving
broadcasts to begin with
to recap the previous commit, in the ward->local authority->county
library we want to return all local authorities and counties. We do this
by excluding anything that doesn't have children.
However, in the countries library, all four countries don't have
children.
I can't think of a generic way to separate these so just filter on the
library id
What was previously ward -> local authority is now a ward -> local
authority -> county. County only covers rural counties and not
metropolitan boroughs and other unitary authorities. Previously, there
was a page full of local authorities (unitary authorities and
districts), and each one of those would have a list of electoral wards.
However, now there are counties that contain a list of districts - so
this needs a new page - a checkbox for "select the county" and then a
list of links to district pages.
If you want to select multiple districts, you'll need to go into each
one of those sub-sections in turn and click select all.
Needed to tweak the query to retrieve the list of areas in a list for a
library. Previously, it just returned anything at top level (ie: didn't
have a parent). However, rural districts now have parents (the rural
counties themselves). So the query now returns "everything that isn't a
leaf node", or in more specific terms, everything that has at least
other row referring to it as a parent. So no electoral wards, since
they dont have any children, but yes to districts and counties.
This commit does two things:
- uses our new polygon-simplifying library to process the polygons
before storing them, rather than processing them in real time
- stores only the polygons in the database, rather than the whole
GeoJSON feature, because we don’t need any of the other information
about the feature
If a library has lots of items then the first 3 should be shown, with
a count of how many more there are, for a total of 4 list items:
> a, b, c, and 23 more
If the library only has 4 items then all 4 should be shown, with
consistent use of conjunction and Oxford comma[1]:
> a, b, c, and d
This keeps the lengths of the examples nice and consistent.
1. We use an Oxford comma because it helps disambiguate when an area
itself has a comma or ‘and’ in it, for example ‘Armagh City, Banbridge
and Craigavon’
When you click through to the page for a library you see the available
areas in alphabetical order. The examples given for each library should
match this.
The given examples should match the choices offered when you visit the
next page. The choices offered on the next page are either the areas
(when a library is not grouped) or the groups (when a library is
grouped).
This commit makes the examples match the choices by excluding sub-areas,
ie those that have a grouping ID.
I think that even with good indexes, querying the area names from one
table is always going to be slow because there’s so much GeoJSON to scan
past.
This commit splits the data into two tables, one for the names and
grouping IDs and one for the blobs of GeoJSON. So for most pages the app
will never even be looking at the table where the GeoJSON is held.
I don’t know if this is a proper, normalised way of structuring the
data, but it does go brrr.
Rather than querying all the features whenever we look up area(s) let’s
only get them when we need them.
The features are really big blobs of data to pass around, so there’s a
significant performance gain to be had from doing this.
