No tests are now using the `client` fixture directly so we can rename
it.
Python convention is to use an `_underscore` for things which should be
considered semi private.
This should discourage people from writing new tests with these old
fixtures.
New tests should always use `client_request`.
Want to be logged in with a different user? Call
`client_request.login(user)` first.
Don’t want to be logged in? Call `client_request.logout()` first (most
of our tests need to be logged in).
Need an instance of `Response` object not an instance of
`BeautifulSoup`? Use `client_request.get_response` or
`client_request.post_response`.
Need to pass in a URL, not arguments to `url_for`? Use
`client_request.get_url(…)` or `client_request.post_url(…)`.
Need to pass in a URL and get a response back? Use
`client_request.get_response_from_url(…)` or
`client_request.post_response_from_url(…)`.
Some tests use the `client` fixture but don’t call any of its methods.
The reason for doing this is because the test depends on something in
the request context.
This commit replaces all those instances with `client_request`, which
also sets the request context.
These tests are the last ones that still use the `client` fixture. By
replacing it with `client_request` we will be able to say that no tests
should be using the `client` fixture directly.
We have a `client_request` fixture which does a bunch of useful stuff
like:
- checking the status code of the response
- returning a `BeautifulSoup` object
Lots of our tests still use an older fixture called `client`. This is
not as good because it:
- returns a raw `Response` object
- doesn’t do the additional checks
- means our tests contain a lot of repetetive boilerplate like `page = BeautifulSoup(response.data.decode('utf-8'), 'html.parser')`
This commit converts all the tests which had a `client.get(…)` or
`client.post(…)` statement to use their equivalents on `client_request`
instead.
Subsequent commits will remove uses of `client` in other tests, but
doing it this way means the work can be broken up into more manageable
chunks.
We added a new argument to `client_request.get` and
`client_request.post` to specify that it should return a raw `Response`
object rather than an instance of `BeautifulSoup`.
This is useful because sometimes we need to look at stuff like the
response headers.
However it turns out we already have a separate method for this, so
rather than invent something new I think it’s better to stick with the
thing we already have.
We have a `client_request` fixture which does a bunch of useful stuff
like:
- checking the status code of the response
- returning a `BeautifulSoup` object
A few of our tests still use an older fixture called
`logged_in_client_with_session`. It’s not clear how this is different
from `logged_in_client`, which we have replaced with `client_request`.
So this commit goes ahead and converts all the tests using
`logged_in_client_with_session` to use `client_request` instead.
We have a `client_request` fixture which does a bunch of useful stuff
like:
- checking the status code of the response
- returning a `BeautifulSoup` object
Lots of our tests still use an older fixture called `logged_in_client`.
This is not as good because:
- it returns a raw `Response` object
- doesn’t do the additional checks
- means our tests contain a lot of repetetive boilerplate like `page = BeautifulSoup(response.data.decode('utf-8'), 'html.parser')`
This commit converts all the tests using `logged_in_client` to:
use `client_request` instead.
We have a `client_request` fixture which does a bunch of useful stuff
like:
- checking the status code of the response
- returning a `BeautifulSoup` object
For most tests of a platform admin view we used `platform_admin_client`
instead. This is not as good because it returns a raw `Response` object
and doesn’t do the additional checks.
This commit converts all the tests using `platform_admin_client` to:
use new `client_request` and log in as `platform_admin_user` before
making any requests.
This is also nice because it makes any test easy to parametrize with
additional users, for example to test differences in behaviour dependant
on being platform admin or not.
Some users have reported a problem with the received text message
report:
> I have tested the reply service but in the excel report the mobile
> number is showing as 4.47900E+23. How can I change the format so that
> it is show the mobile number that has replied?
This is happening because Excel is interpreting a phone number in the
format `447900900123` as a number in
[scientific notation](https://en.wikipedia.org/wiki/Scientific_notation),
in other words 4.479 × 10<sup>23</sup>.
`447900900123` is the format that our provider is giving us the number
in – there’s no guarantee it will always be in this format.
We can prevent this behaviour by putting spaces in the numbers. Excel
and Google Sheets won’t try to convert a string with spaces into a
number.
I think we used to do this for the sent text messages report but
probably stopped because we decided it was better to keep the phone
number in the same format as it had been supplied to us for
reconcilliation purposes.
This makes it clearer that this model collection isn’t the organisations
for a user or a service or some other entity, like most model
collections are.
It will also lets us make a separate Organisations model, without the
name conflicting.
the api returns UTC timestamps, we should keep them as UTC timestamps
until the very last moment, and only convert them into BST when we know
we want to return to a user (ie: in contact-list.html and other places
like that)
Previously we were passing a flag to the API which handled this. Now
we are doing it at the time of clicking the link, not at the time of
storing the new password. We don’t need to update the timestamp twice,
so this commit removes the code which tells the API to do it.
This wasn't adding anything now that we have two new and more specific
fixtures, `active_user_create_broadcasts_permission` and
`active_user_approve_broadcasts_permission`, that can be used instead.
`manage_templates` has now been removed from the `create_broadcasts`
permission, so this also adjusts the fixture for a user who can create
broadcasts.
The `send_messages` permission has been deprecated for use with
broadcast services, so we can drop support for it in the code. We
were supporting both the old permissions and new permissions
(`create_broadcasts` and `approve_broadcasts`) while we switched people
over.
This removes `send_messages` from the `user_has_permissions` decorator
around the broadcast routes and from the page to view a broadcast and
broadcast dashboards. We can now git rid of a lot of the parameterization
that was temporarily added to the tests.
We've added new broadcast roles in the database (`create_broadcasts` and
`approve_broadcasts`).
Adding these has meant we've needed to do a bit of a rewrite of the roles and
permissions code since this had been based on the assumption that each
database permission only belongs to one admin role - this is no longer true.
This means that flipping the roles dict round to create a dict which
contains database permissions as the keys is no longer possible. We can't
necessarily tell which admin role someone has given a database permission.
To check if a user has an admin role given a list of database permissions,
the user must now have ALL the database permissions mapped to that role
(instead of just one). This works because no one has the `manage_users`
permission without also having the `manage_settings` (and similar for
the other admin roles which map to multiple database permissions).
Some test data was changed because it was using admin roles where
database permissions are actually used when the app is running. I've kept
the functionality of the `translate_permissions_from_db_to_admin_roles`
function passing through any unknown roles it is passed as an argument.
This is not necessary, so can be changed later if we decide it will not
ever be used. However, removing it would require updating a lot of
tests since the tests rely on this behaviour.
Added two new permissions - `create_broadcasts` and
`approve_broadcasts`. These new permissions get added to the
`has_permissions` decorator of the broadcast routes to allow the routes
to be accessed with either the old permissions on the new ones while we
switch over.
We were using the `send_messages` permission for the broadcast routes.
By having two new permissions we can allow a more granular control of
these routes.
If a job exceeds the daily sending limit, show that on the job page. The job is only created if the sending limit has been reached when the delivery app is processing the job, usually this error is caught at the time the CSV is uploaded and the job is not created.
This allows us to roll out the feature to other users. Note that
the flag is also "True" if the user has "webauthn_auth" as their
auth type, so this is compatible with the more fine-grained check
we have on the authentication parts of the feature. We could do a
more explicit "can_use_webauthn or webauthn_auth" check here, but
the idea is that we'll be able to get rid of this flag eventually,
so I've optimised for brevity instead.
I've modified a couple of the unhappy-path tests to make it more
explicit that the flag is false, since it can be true for Platform
Admins and "normal users" alike.
this is a bit complex, but essentially we're using the test variables
defined in the duolabs py_webauthn library [1]. We're already using
their test variables in tests/app/models/test_webauthn_credential.py and
in the webauthn_credential fixture in conftest.py. By using sample
signature, authenticatordata and clientdatajson from the same key we can
test that the library correctly verifies the signed challenge matches
the original.
We needed to transform some of this data as the yubico/fido2 library we
use has a slightly different way of formatting the fields for the
request body, which is why we're doing things like base64 decoding and
converting from hex to bytes in the post data.
The pytest fixture has changed - before it was incomplete/corrupted and
would error when trying to verify the signature. We took the
credential_data from the pytest fixture, converted it to an
AttestedCredentialData using WebauthnCredential.to_credential_data,
modified the public_key private dictionary to add `public_key[-1]: 1`,
and then called `AttestedCredentialData.create` to re-CBOR-encode the
blob.
The `-1: 1` is the numeric ID of the "SECP256R1" elliptic curve
algorithm. The py_webauthn library forces this particular algorithm,
which differs from the sample creds we took from the fido2 lib tests,
which is why we've had to update our data.
[1] https://github.com/duo-labs/py_webauthn/blob/master/tests/test_webauthn.py#L13-L32
This naming was introduced in 2016 without explanation [1]. I find it
confusing because:
- It's reminiscent of "_app", which is a Python convention indicating
the variable is internal, so maybe avoid using it.
- It suggests there's some other "app" fixture I should be using (there
isn't, though).
The Python style guide describes using an underscore suffix to avoid
clashes with inbuilt names [1], which is sort of applicable if we need
to import the "app" module [2]. However, we can also avoid clashes by
choosing a different name, without the strange underscore.
[1]: 3b1d521c10
[2]: 78824f54fd/tests/app/main/views/test_forgot_password.py (L5)
We need to re-initialise the webauthn_server module with original
app config, since this state is global across all tests. Since the
behaviour of the original fixture wasn't specific to verifying the
origin, I've renamed the fixture as part of making it global.
In order to keep the fixture simple, I've rewritten the test for
the webauthn_server module, so they don't touch the app fixture.
This links up the `get_webauthn_credentials_for_user` and
`create_webauthn_credential_for_user` methods of the user api client to
notifications-api.
To send data to the API we need strings to be unicode, so we call
decode('utf-8') on base64 objects.
Co-authored-by: Leo Hemsted <leo.hemsted@digital.cabinet-office.gov.uk>
This passes existing credentials in the server response, to allow
the browser to prevent re-registering the same key for the same
user. Registering the same key multiple times doesn't seem to be
an issue technically; the user has likely got their keys mixed up.
- Chrome says "you don't need to register it again".
- Safari exits with an InvalidStateError.
- Firefox exits with a DOMException.
This adds Yubico's FIDO2 library and two APIs for working with the
"navigator.credentials.create()" function in JavaScript. The GET
API uses the library to generate options for the "create()" function,
and the POST API decodes and verifies the resulting credential. While
the options and response are dict-like, CBOR is necessary to encode
some of the byte-level values, which can't be represented in JSON.
Much of the code here is based on the Yubico library example [1][2].
Implementation notes:
- There are definitely better ways to alert the user about failure, but
window.alert() will do for the time being. Using location.reload() is
also a bit jarring if the page scrolls, but not a major issue.
- Ideally we would use window.fetch() to do AJAX calls, but we don't
have a polyfill for this, and we use $.ajax() elsewhere [3]. We need
to do a few weird tricks [6] to stop jQuery trashing the data.
- The FIDO2 server doesn't serve web requests; it's just a "server" in
the sense of WebAuthn terminology. It lives in its own module, since it
needs to be initialised with the app / config.
- $.ajax returns a promise-like object. Although we've used ".fail()"
elsewhere [3], I couldn't find a stub object that supports it, so I've
gone for ".catch()", and used a Promise stub object in tests.
- WebAuthn only works over HTTPS, but there's an exception for "localhost"
[4]. However, the library is a bit too strict [5], so we have to disable
origin verification to avoid needing HTTPS for dev work.
[1]: c42d9628a4/examples/server/server.py
[2]: c42d9628a4/examples/server/static/register.html
[3]: 91453d3639/app/assets/javascripts/updateContent.js (L33)
[4]: https://stackoverflow.com/questions/55971593/navigator-credentials-is-null-on-local-server
[5]: c42d9628a4/fido2/rpid.py (L69)
[6]: https://stackoverflow.com/questions/12394622/does-jquery-ajax-or-load-allow-for-responsetype-arraybuffer
This adds a new platform admin settings row, leading a page which
shows any existing keys and allows a new one to be registered. Until
the APIs for this are implemented, the user API client just returns
some stubbed data for manual testing.
This also includes a basic JavaScript module to do the main work of
registering a new authenticator, to be implemented in the next commits.
Some more minor notes:
- Setting the headings in the mapping_table is necessary to get the
horizontal rule along the top (to match the design).
- Setting caption to False in the mapping_table is necessary to stop
an extra margin appearing at the top.
Do not allow platform admins to:
- create broadcasts
- approve broadcasts
- reject broadcasts
that is, unless they have a send_messages permission
for a given service.
This is so platform admins have the minimum permissions necessary
to cancel a broadcast that might have been sent out accidentally.
Two reasons to not hide rejected broadcasts:
- if a broadcast was rejected by mistake then it’s useful to have an
audit of who did that
- it means you can still see old broadcasts without having to leave
in pending-approval, which is dangerous because they might
accidentally be approved
The `/accounts` page was listing trial mode services twice if a user
belonged to an org. They were shown under both the 'Live services' and
'Trial mode services' sections. After this change, 'Live services' will
show all live services (whether or not they belong to an org) and 'Trial
mode services' will show all trial mode services. If a user belongs to an
org, they will also see the summary of how many services per org at the
top of the page.
A couple of services in tests were renamed for clarity.
first of a two step process to remove invited user objects from the
session. we're removing them because they're of variable size, and with
a lot of folder permissions they can cause the session to exceed the 4kb
cookie size limit and not save properly.
this commit looks at invited org users only.
in this step, start saving the invited org user's id to the
session alongside the session object. Then, if the invited_org_user_id
is present in the next step of the invite flow, fetch the user object
from the API instead of from the session. If it's not present (due to a
session set by an older instance of the admin app), then just use the
old code to get the entire object out of the session.
For invites where the user is small enough to persist to the cookie,
this will still save both the old and the new way, but will always make
an extra check to the API, I think this minor performance hit is totally
fine. For invites where the user is too big to persist, they'll still
fail for now, and will need to wait until the next PR comes along and
stops saving the large invited user object to the session entirely.
We think that in some cases alerts will be composed in the moment, and
therefore making people first create a template is:
- not a good use of their time
- adding some conceptual complexity which they don’t need
This commit makes it possible to type some words and have them go
straight into the `content` field in the database.
In the future we might want to progressively enhance the radio buttons
so they show on the same page (like we do with the grey buttons on the
templates page).
We shouldn’t have a page where someone can look up any other user’s
email address based on their user ID.
We also don’t want a page where a malicious user could send someone an
link which would get them invited to the service.
Restricting the invite to be populated just from users in their own
organisation doesn’t mitigate against this stuff completely, but they
probably have a way of finding out the email address of someone in their
organisation already.
When we get a support ticket we need to check whether a user has any
live services.
We have a method for this on the user model now, so we don’t need a
separate function in the feedback code.
It wasn’t very well tested so I’ve adapted the old tests from the
feedback view to work against the method on the user model too.
