I emailed the Geography team at the ONS:
> Hi geography team,
>
> I work on GOV.UK Notify, which is a service run by Government Digital Service (part of the Cabinet Office). I was given your email address by [redacted] who’s been helping answer some of my questions on the cross-government Slack.
>
> We’re using some of the boundary datasets from the Open Geography Portal, and mostly they’ve been excellent.
>
> In the abstract, the problem we’re trying to solve is, given a point outside an area, what is the minimum distance to a point within that area. So, for example, if a crow was somewhere in Cardiff, what’s the shortest distance it would have to fly to reach somewhere in the Bristol local authority district?
>
> We’ve noticed some problems with the data that means our calculations would be wrong. We’ve noticed this around Torquay, Norwich and Bristol. Here are some screenshots of Bristol, from the generalised and full resolution boundaries:
>
> The artefacts I’ve highlighted are closer to Cardiff than any actual part of the land area of Bristol. They are either:
> - in the sea
> - land that’s part of North Somerset
>
> I suspect that this is being caused by the process of clipping the actual region of Bristol (which, unusually, extends into the water) to the mean high water line.
>
> I’ve worked around this by filtering out any polygons that are smaller than ~7,500m². It’s a bit hacky because parts of the Scilly Isles start disappearing. That’s not a problem for what I’m working on, but it would be nice to not need the hack.
>
> So my questions would be:
>
> - Is there a better way to remove these artefacts than filtering by area?
> - Is there a plan to remove these artefacts from the data in future releases?
>
> Thanks in advance,
> Chris
They emailed back to say:
> Hi Chris
>
> Thank you for your enquiry.
>
> We have completed the amendments to the LAD MAY 2020 BFC and BGC boundaries as mentioned so you should be able to download them from the portal now.
>
> Hope this helps.
>
> Kind regards
> [redacted]
This commit brings in the files they’ve updated. We still have to do
some filtering (but now at a higher resolution) because they haven’t
fixed Norwich yet. I’ll email them separately about that.
If you’re adding another area to your broadcast it’s likely to be close
to one of the areas you’ve already added.
But we make you start by choosing a library, then you have to find the
local authority again from the long list. This is clunky, and it
interrupts the task the user is trying to complete.
We thought about redirecting you somewhere deep into the hierarchy,
perhaps by sending you to either:
- the parent of the last area you’d chosen
- the common ancestor of all the areas you’d chosen
This approach would however mean you’d need a way to navigate back up
the hierarchy if we’d dropped you in the wrong place. And we don’t have
a pattern for that at the moment.
So instead this commit adds some ‘shortcuts’ to the chose library page,
giving you a choice of all the parents of the areas you’ve currently
selected. In most cases this will be one (unitary authority) or two
(county and district) choices, but it will scale to adding areas from
multiple different authorities.
It does mean an extra click compared to the redirect approach, but this
is still fewer, easier clicks compared to now.
This meant a couple of under-the-hood changes:
- making `BroadcastArea`s hashable so it’s possible to do
`set([BroadcastArea(…), BroadcastArea(…), BroadcastArea(…)])`
- making `BroadcastArea`s aware of which library they live in, so we can
link to the correct _Choose area_ page
At the moment there are some areas which have:
- a `count_of_phones` value of `None`
- no sub-areas
This is wrong, but until we fix the data the phone counting code needs
to handle this.
This commit:
- adds the `or 0` in the right place (where it will catch these areas
with missing data)
- adds a test which checks these areas, and compares them to other kinds
of areas
This is a better name for the module because it’s:
- not just constants, there’s a method in here now
- only stuff to do with populations, not other kinds of constants
We need to give people a better feel for the consequences of
broadcasting an alert. We’ve seen in research that some users will
assume it is subscription based, or opt-in, rather than going to every
phone in the area.
I reckon that the most effective way to communicate this is to put some
numbers next to the areas, to give people an idea of how many people
will get alerted.
We can estimate how many phones are in an area by:
- taking the population of all electoral wards in that area
- multiplying it by the percentage of people who own an internet
connected phone[1]
The Office for National Statistics publish both these datasets.
The number of people who own an intenet connected phone varies a lot by
age. Since the population data for each ward is broken down by age we
can factor this in. Simplified, the calculation looks like this:
- take the _Abbey_ ward of _Barking and Dagenham_
- in this ward there are 26 people aged 80
- 40% of people over 65 have an internet-connected phone
- therefore 10 of these 80-year-olds would be likely to receive a
broadcast
- (repeat for all other ages)
These numbers won’t be exact, but should be enough to give people a feel
for the severity of what they’re about to do. We can see if they acheive
this aim in user research.
1. This is a proxy for the number of people who are likely to have a 4G
capable phone, because only 4G capable phones will be receiving
broadcasts to begin with
We filter out very small polygons from the original data to remove
glitches. These glitches are caused by trying to subtract the water from
a polygon that includes some land and some water, but using two
different definitions or resolutions of mean high water line.
If we don’t do this then we end up with a bunch of very small polygons
which lie far outside the understood area of a place, causing large
overspill.
We need to increase the threshold for this process because we’re still
seeing this problem around Bristol and Norwich.
This does mean we lose a few very small polygons in places like Shetland
and the Scilly Isles, but not in such a way that we would avoid
broadcasting to them (because they’d still be caught by the
simplification and overspill).
to recap the previous commit, in the ward->local authority->county
library we want to return all local authorities and counties. We do this
by excluding anything that doesn't have children.
However, in the countries library, all four countries don't have
children.
I can't think of a generic way to separate these so just filter on the
library id
What was previously ward -> local authority is now a ward -> local
authority -> county. County only covers rural counties and not
metropolitan boroughs and other unitary authorities. Previously, there
was a page full of local authorities (unitary authorities and
districts), and each one of those would have a list of electoral wards.
However, now there are counties that contain a list of districts - so
this needs a new page - a checkbox for "select the county" and then a
list of links to district pages.
If you want to select multiple districts, you'll need to go into each
one of those sub-sections in turn and click select all.
Needed to tweak the query to retrieve the list of areas in a list for a
library. Previously, it just returned anything at top level (ie: didn't
have a parent). However, rural districts now have parents (the rural
counties themselves). So the query now returns "everything that isn't a
leaf node", or in more specific terms, everything that has at least
other row referring to it as a parent. So no electoral wards, since
they dont have any children, but yes to districts and counties.
they're in both the local authority and County & Unitary Authority data
sets, so we should only add them once. Keep the LA one so the ctyua19
dataset is only used for counties
map lower tier local authorities (districts within rural counties) to
upper tier local authorities (administrative counties) using the new
ctyua19 dataset.
nb: unitary authorities (eg: Southwark) are both lower tier AND upper
tier. For this first pass they turn up twice in the db, eg Southwark is
in there as `lad20-E09000028` and `ctyua19-E09000028`. For now, the
electoral wards within Southwark are mapped to the pre-existing
`lad20-E09000028`. Both lad20 and ctyua19 have data sets in
broadcast_area_features.
maps lower tier local authorities to upper tier local authorities -
translation for humans: Maps districts to the counties that they are in.
For counties, a row looks like:
`255,E07000105,Ashford,E10000016,Kent`
(E07000105=Ashford, E10000016=Kent)
For unitary districts it maps that district to itself, eg:
`49,E06000052,Cornwall,E06000052,Cornwall`
where both codes are the same
Since the key relies on visual association between the shapes on the
maps and the styling of the key, it won’t work for non-visual users.
An alternative way of giving them the same information is by providing
the size of the area numerically.
We have a bunch of stuff for doing lat/long transformation in the
`BroadcastMessage` class. This is not a good separation of concerns, now
that we have a separate class for dealing with polygons and coordinates.
This commit does two things:
- uses our new polygon-simplifying library to process the polygons
before storing them, rather than processing them in real time
- stores only the polygons in the database, rather than the whole
GeoJSON feature, because we don’t need any of the other information
about the feature
Simplifying polygons means reducing the number of points used to render
them. This commit implements simplification such that, for any given
input polygons, the combined point count of the simplified polygons is
less than 100.
When simplifying the polygons we are trying to get the smallest number
of points while meeting these two rules:
1. No part of the area the user has chosen can be cut off
2. The area of the simplified polygon should be as small as possible
This commit introduces two techniques we weren’t using before:
1. Dilating and eroding the area to fill in concave details of the
shape, like inlets and harbours[1]
2. Making the simplification threshold proportionate to the perimeter of
all polygons, so bigger and crinklier polygons get more
simplification applied
It also shows the estimated bleed as a separate polygon. This lets us
make it bigger (so it’s more closer the the approximate bleed) without
having to send a bigger area to the CBC and compounding the amount of
actual bleed.
1. Inspired by this blog post about ‘removing the crinkley bits’ from
Vancouver Island:
http://blog.cleverelephant.ca/2010/11/removing-complexities.html
It’s been superceded by the ‘Local’ library (formerly ‘Electoral wards
in the United Kingdom’).
The latter is better because:
- it’s covers all 4 nations, not just England and Wales
- it has electoral wards as well as local authorities which group them,
so there’s more flexibility when choosing an area to broadcast to
We’ve observed people using ‘national’ and ‘local’ during user research.
It has less tongue-twisting ambiguity than county vs country.
But we think that maybe just getting rid of ‘counties’ is enough to
disambiguate them. So this commit just takes the ‘local’ concept.
This commit also gives the libraries and areas new IDs, which means if
we want to rename them in the future it won’t be a breaking change.
Broadcasting is not a precise technology, because:
- cell towers are directional
- their range varies depending on whether they are 2, 3, 4, or 5G
(the higher the bandwidth the shorter the range)
- in urban areas the towers are more densely packed, so a phone is
likely to have a greater choice of tower to connect to, and will
favour a closer one (which has a stronger signal)
- topography and even weather can affect the range of a tower
So it’s good for us to visually indicate that the broadcast is not as
precise as the boundaries of the area, because it gives the person
sending the message an indication of how the technology works.
At the same time we have a restriction on the number of polygons we
think and area can have, so we’ve done some work to make versions of
polygons which are simplified and buffered (see
https://github.com/alphagov/notifications-utils/pull/769 for context).
Serendipitously, the simplified and buffered polygons are larger and
smoother than the detailed polygons we’ve got from the GeoJSON files. So
they naturally give the impression of covering an area which is wider
and less precise.
So this commit takes those simple polygons and uses them to render the
blue fill. This makes the blue fill extend outside the black stroke,
which is still using the detailed polygons direct from the GeoJSON.
It made for a good early demo to show how we could have different
libraries, but we’d don’t think there’s a strong user need for being
able to broadcast to a region of England.
Regions also have the problem that:
- they are ambiguous – both England and Scotland have a region called
‘South east’
- Northern Ireland doesn’t have formal regions
This commit removes the regions library.
If a library has lots of items then the first 3 should be shown, with
a count of how many more there are, for a total of 4 list items:
> a, b, c, and 23 more
If the library only has 4 items then all 4 should be shown, with
consistent use of conjunction and Oxford comma[1]:
> a, b, c, and d
This keeps the lengths of the examples nice and consistent.
1. We use an Oxford comma because it helps disambiguate when an area
itself has a comma or ‘and’ in it, for example ‘Armagh City, Banbridge
and Craigavon’
When you click through to the page for a library you see the available
areas in alphabetical order. The examples given for each library should
match this.
The given examples should match the choices offered when you visit the
next page. The choices offered on the next page are either the areas
(when a library is not grouped) or the groups (when a library is
grouped).
This commit makes the examples match the choices by excluding sub-areas,
ie those that have a grouping ID.
Now that the data needed to create a `BroadcastArea` is pretty
lightweight because it doesn’t include the GeoJSON we can go back to
putting it in memory when we start up the app, to make the pages load
really fast.
Rough estimate for the size of this dataset:
> 10,000 areas
> Average length of area name = 20 characters
> Average length of area id = 20 characters
> Size of one area in bytes = 20 + 20 = 40
> Size of dataset = 40 * 10,000 = 400,000 bytes = 400kb
I think that even with good indexes, querying the area names from one
table is always going to be slow because there’s so much GeoJSON to scan
past.
This commit splits the data into two tables, one for the names and
grouping IDs and one for the blobs of GeoJSON. So for most pages the app
will never even be looking at the table where the GeoJSON is held.
I don’t know if this is a proper, normalised way of structuring the
data, but it does go brrr.
Rather than querying all the features whenever we look up area(s) let’s
only get them when we need them.
The features are really big blobs of data to pass around, so there’s a
significant performance gain to be had from doing this.
