This will capture and send various events to Sentry:
- Any unhandled exceptions.
- Any logger.error calls.
- Some request traces.
The latter are severely limited to avoid going over the free tier
limits for Sentry, and to avoid excess effort on our end.
While the package can always fetch new holidays via the GOV.UK API,
the latest version of the packages also caches ones for next year,
which means we can avoid unnecessary web requests.
This brings a few performance improvements for RecipientCSV, which
we use to preview and process CSVs. One change also renames one of
the attributes for the class to "guestlist".
Makes the mock up of an alert we show use an
inline SVG instead of it as a background image.
This means it can use the colour of the heading
text next to it in a way that adapts when high
contrast mode is on.
https://github.com/alphagov/notifications-utils/pull/922
Brings in https://github.com/alphagov/notifications-utils/pull/889/files
At the moment, we are not doing any transformation of features before
applying geometric algorithms to them. This is, in effect, assuming that
the earth is flat.
This new version of utils implements the transformation of our polygons
to a Cartesian plane. In other words, it converts them from being
defined in spherical degrees to metres.
For the admin app this means we need to convert places where the code
expects things to be measured in degrees to work in metres instead.
WTForms versions less than 3.0.0 have a security vulnerability where
arbitrary HTML can be inserted into the label of a form, allowing the
possibility of a cross-site scripting attack.
I don’t know if there’s anywhere we put user-generated content into form
labels but it’s possible we are vulnerable somewhere.
This require moving some imports because as of
https://github.com/wtforms/wtforms/pull/614/files
there is no longer a separate module for HTML 5 fields, they are now
considered core fields.
As of https://github.com/wtforms/wtforms/issues/445/files custom
implementations of `pre_validate` or `post_validate` must raise
`ValidationError` to trigger a validation message, where we were raising
`ValueError` this was no longer being caught.
As of https://github.com/wtforms/wtforms/pull/355/files `StringField`
returns `None` for empty data, not `''` but our `validate_email_address`
function only accepts strings.
It’s a pain having to remember to update both of them. Looks like `pip`
now supports a different syntax without the `egg` bit which means a git
dependency only needs the version number changing in one place.
Previously when trying to bring in
https://github.com/alphagov/notifications-utils/pull/889 we had trouble
because there were no wheels available for the version of Pip that we
were using. Now that we have upgraded to Python 3.9 we are using a
newer buildpack which should have the necessary wheels available.
This commit is just installing the package and nothing else, to give us
more confidence that we can merge
https://github.com/alphagov/notifications-utils/pull/915/files without
having to subsequently revert it.
We can’t use the latest version of importlib-metadata because it’s
pinned to <4.3 by the newest version of flake8.
The conflict is caused by:
The user requested importlib-metadata==4.8.1
click 8.0.1 depends on importlib-metadata; python_version < "3.8"
pytest 6.2.5 depends on importlib-metadata>=0.12; python_version < "3.8"
flake8 4.0.1 depends on importlib-metadata<4.3; python_version < "3.8"
Our `make freeze-requirements` task doesn’t catch this because it
doesn’t look at dependencies in `requirements-for-test.txt`. Therefore
it only freezes the version that `click` is specifying, which is the
latest version.
Pinning the version in `requirements.in` gets around this.
Flask version 2 requires Jinja2 version 3. However rather than bumping
both at once it feels safer to do this incrementally, so we can isolate
anything which breaks.
This is the newest version.
Pyup is complaining about vulnerabilities in version 1.0.1, specifically
> Werkzeug version 2.0.2 improves the security of the debugger cookies.
> "SameSite" attribute is set to "Strict" instead of "None", and the
> secure flag is added when on HTTPS.
Previously we were using whatever version of Werkzeug that Flask
specified this pins it to get rid of the vulnerability without having to
upgrade everything at once.
This requires a few changes to tests which were relying on importing
`session` and `current_user` from Flask. Previously it seemed that
importing these in the tests referred to the same object that was being
used in the app. This appears to no longer be the case. This commit
works around that by:
- using a context manager to get the contents of the session, like we
already do in most tests
- asserting that the mock which logs the user in is being called with
the right values, rather than looking at the state of the
`current_user` object (which was probably giving false certainty
anyway)
This adds Yubico's FIDO2 library and two APIs for working with the
"navigator.credentials.create()" function in JavaScript. The GET
API uses the library to generate options for the "create()" function,
and the POST API decodes and verifies the resulting credential. While
the options and response are dict-like, CBOR is necessary to encode
some of the byte-level values, which can't be represented in JSON.
Much of the code here is based on the Yubico library example [1][2].
Implementation notes:
- There are definitely better ways to alert the user about failure, but
window.alert() will do for the time being. Using location.reload() is
also a bit jarring if the page scrolls, but not a major issue.
- Ideally we would use window.fetch() to do AJAX calls, but we don't
have a polyfill for this, and we use $.ajax() elsewhere [3]. We need
to do a few weird tricks [6] to stop jQuery trashing the data.
- The FIDO2 server doesn't serve web requests; it's just a "server" in
the sense of WebAuthn terminology. It lives in its own module, since it
needs to be initialised with the app / config.
- $.ajax returns a promise-like object. Although we've used ".fail()"
elsewhere [3], I couldn't find a stub object that supports it, so I've
gone for ".catch()", and used a Promise stub object in tests.
- WebAuthn only works over HTTPS, but there's an exception for "localhost"
[4]. However, the library is a bit too strict [5], so we have to disable
origin verification to avoid needing HTTPS for dev work.
[1]: c42d9628a4/examples/server/server.py
[2]: c42d9628a4/examples/server/static/register.html
[3]: 91453d3639/app/assets/javascripts/updateContent.js (L33)
[4]: https://stackoverflow.com/questions/55971593/navigator-credentials-is-null-on-local-server
[5]: c42d9628a4/fido2/rpid.py (L69)
[6]: https://stackoverflow.com/questions/12394622/does-jquery-ajax-or-load-allow-for-responsetype-arraybuffer
The Python rtree library we are using to build RTrees has a dependency
on the C package libspatialindex. This package is not installed on PaaS,
so it’s hard for us to use it.
This commit changes the code to use a library called rtreelib instead.
rtreelib doesn’t have a built in way to serialise the index it builds,
so I’ve had to implement that using pickle.
We want to know how many phones are in a user-supplied polygon, so we
can show the impact of a broadcast, in the same way that we do when
users pick areas from our library.
We already know how many phones are in each electoral ward. But there
are challenges with an arbitrary polygon:
- where it does overlap a ward, the overlap could be partial
- it could overlap more than one ward
- finding out which wards it overlaps by brute force (looping through
all the wards and seeing which ones intersect with our polygon) would
be way to slow to do in real time
Instead we can use a data structure called an R-tree[1] to build an
index which provides a much, much faster way of looking up which
polygons overlap another. We can build this tree in advance and save it
somewhere, which means there’s a lot of computation we don’t need to do
in real time.
The R-tree returns a set of objects (ward IDs) which we can go and look
up in our library of electoral wards. These wards will be the ones that
might have some overlap with our custom polygon.
Once we have this small set of wards which might overlap our ward, we
can look at the size of the area of overlap (relative to the size of the
whole ward) and multiply that by the known count of phones in that ward
to get an approximation of the count of phones in the overlap area.
Summing these approximations give an estimate for the whole area of the
custom polygon.
1. https://en.wikipedia.org/wiki/R-tree
we saw exceptions on prod that I think might have caused a worker that
is being terminated to die ungracefully. While I'm not sure if this is
an actual problem that changed behaviour (app instances crashing and
restarting), at the very least it definitely polluted the logs and
obscured any actual issues we were having.
https://github.com/prometheus/client_python/releases/tag/v0.10.0
see the pending fix for this problem here:
https://github.com/prometheus/client_python/pull/644
