Commit Graph

3688 Commits

Author SHA1 Message Date
Chris Hill-Scott
2accf8434a Remove false precision from area estimates
We give estimates of the area for those who can’t see the map. These
estimates were needlessly precise, gave a false sense of accuracy and
were causing intermittent test failures between different environments.

This commit rounds them in the same way that we round the count of
phones.
2021-07-06 17:00:51 +01:00
Chris Hill-Scott
e4ca78634d Bump utils to bring in new polygon simplification
We’ve changed our simplification a bit so:
- polygons have slightly more points (see https://github.com/alphagov/notifications-utils/pull/873)
- the individual points have less precision (see https://github.com/alphagov/notifications-utils/pull/872)

Overall this reduces the size of the data we’re storing from 74MB to
63MB, and should make any pages where we are rendering lots of
coordinates load a bit quicker.
2021-07-06 17:00:50 +01:00
Leo Hemsted
416b5c3e26 Merge pull request #3926 from alphagov/sign-in-bug
ensure user details are always in the session after entering password
2021-07-06 11:56:24 +01:00
Chris Hill-Scott
960d2b8bf1 Merge pull request #3954 from alphagov/no-empty-state-choose-areas
Remove empty state step before choosing areas
2021-07-05 16:01:49 +01:00
Chris Hill-Scott
2cfd22b20c Remove empty state step before choosing areas
We had some kind of idea that having this empty page would introduce the
idea of choosing areas and reinforce that you are building up a list of
areas.

But since the journey is now so simple with the button to create an
alert directly on the dashboard page, maybe people don’t need this extra
orientation.
2021-07-05 14:33:02 +01:00
Leo Hemsted
74b5d8f3cd add test for when trying to sign in as another invited user 2021-07-02 18:11:53 +01:00
Chris Hill-Scott
5a378fe51f Use CustomBroadcastArea to estimate phones in bleed area
Our current assumption is that the bleed area has the same population
density as the broadcast area.

This is particularly naïve when:
- the bleed area overlaps the sea – no-one lives in the sea
- the broadcast area is a village and the bleed area is the surrounding
  countryside
- the broadcast area is adjacent to a densely populated area like a city

We can be smarter about this now that we have a way of determining the
number of phones in an arbitrary area, based on the known areas that we
have population data about.

Calculating the population in an overlap is a slightly more intensive
calculation. So we only doing it for areas which are smaller enough that
it doesn’t slow things down too much. For larger areas we still use the
more naïve algorithm.
2021-07-02 10:36:25 +01:00
Ben Thorner
1f33924ceb Send upload_id to Template Preview for logging
This means we can include the anonymous ID for the file in the log
we have about Type3 fonts [1]. Currently, we have no way of tracing
manually uploaded files with this potential defect.

[1]: https://github.com/alphagov/notifications-template-preview/pull/557
2021-07-01 12:09:47 +01:00
Leo Hemsted
7b3751240c ensure user details are always in the session after entering password
We signal that we're mid-way through the sign-in flow by adding a
`user_details` dict to the session.

previously, we'd only put a user's details in the session in `User.sign_in`,
just before sending any 2fa prompt and redirecting to the two factor
pages.

However, we found a bug where a user with no session (eg, using a fresh
browser) tried to log in, but they had never clicked the link to
validate their email address when registering. Their user's state was
still in "pending", so we redirected to `main.resend_email_verification`
as intended - however, they didn't have anything in the session and the
resend page expected to get the email address to resend to out of that.

To be safe, as soon as we've confirmed the user has entered their
password correctly, lets save the session data at that point. That way
any redirects will be fine.
2021-06-29 18:13:25 +01:00
Leo Hemsted
71613dd942 remove old /two-factor endpoint and update test names
we redirect people to `/two-factor-sms` since #26ad20719
2021-06-29 16:19:24 +01:00
Leo Hemsted
c439cafd37 always create platform admin users with webauthn in tests 2021-06-29 16:19:22 +01:00
Leo Hemsted
126f9cf6be fix bug stopping editing of permissions of webauthn platform admins
We hide the radio field in the HTML for platform admins, as we don't
want anyone to be able to change their auth type. However, when the form
is validated, the form has a field called login_authentication that it
expects a value for. It silently fails as it complains that when the
user POSTed they didn't select a value for that radio field, but the
error message is on the radio fields that don't get displayed to the
user so they'd never know.

Fixing this is actually pretty hard.

We use this form in two places, one where we have a user to edit, one
where we are creating an invite from scratch. So sometimes we don't know
about a user's auth type. In addition, radio buttons are mandatory by
design, but now sometimes we don't just want to make it optional but
explicitly ignore the value being passed in? To solve this, remove the
field entirely from the form if the user is a platform admin. This means
that if the code in manage_users.py tries to access the
login_authentication value from the form, it'll error, but I think
that's okay to leave for now given we concede that this isn't a perfect
final solution.

The tests didn't flag this previously as they tried to set from sms_auth
(the default for `platform_admin_user`) TO email_auth or sms_auth. Also,
the diagnosis of this bug was confounded further by the fact that
`mock_get_users_by_service` sets what is returned by the API - the
service model then takes the IDs out of that response and calls
`User.get_user_by_id` for the matching ID (as in, the code only uses
get_users_by_service to ensure the user belongs to that service). This
means that we accidentally set the form editing the current user, as
when we log in we set `get_user_by_id` to return the user of our choice
2021-06-29 15:53:48 +01:00
Leo Hemsted
92b6885224 ensure webauthn page aborts if user isn't allowed 2021-06-29 15:53:48 +01:00
Rebecca Law
5534ecb5a4 Merge pull request #3939 from alphagov/check-daily-limit-for-csv-uploads
Check the daily limit get the daily notification_count from redis.
2021-06-29 14:41:22 +01:00
Rebecca Law
77c2aa9fd6 Stop passing in today_only for the get_service_statistics method.
We now only ever call it with False.
To remove it from the api call will require a change in the API so will do that at another time.
2021-06-29 07:33:40 +01:00
Ben Thorner
e7c1ce1869 Merge pull request #3938 from alphagov/save-original-pdf-uploads
Backup original precompiled uploads to S3
2021-06-24 12:43:40 +01:00
Ben Thorner
b3f48c1a84 Backup original precompiled uploads to S3
This continues the work from Template Preview [1], so that we have
a complete store of original PDFs to use for testing changes to it.

Previously we did store some originals, but these were only invalid
PDFs that had failed sanitisation; for valid PDFs, the "transient"
bucket only contains the sanitised versions, which the API deletes
/ moves when the notification is sent [2].

Since the notification is only created at a later stage [3], there's
no easy way to get the final name of the PDF we send to DVLA. Instead,
we use the "upload_id", which eventually becomes the notification ID
[4]. This should be enough to trace the file for specific debugging.

Note that we only want to store original PDFs if they're valid (and
virus free!), since there's no point testing changes with bad data.

[1]: https://github.com/alphagov/notifications-template-preview/pull/545
[2]: c44ec57c17/app/service/send_notification.py (L212)
[3]: 7930a53a58/app/main/views/uploads.py (L362)
[4]: 7930a53a58/app/main/views/uploads.py (L373)
2021-06-24 12:21:05 +01:00
Katie Smith
1862cd0223 Update data format for platform admin report
The format of dates returned from API for the "Monthly notification
statuses for live services" report has changed, so we no longer need to
format the dates here.

https://github.com/alphagov/notifications-api/pull/3279
2021-06-23 16:14:58 +01:00
Rebecca Law
44f02f2e30 To check the daily limit get the daily notification_count from redis.
The daily limit cache is set by the api when a notification is created. There is one cache key per service per day and it expires after 24 hours.
2021-06-23 15:56:05 +01:00
Pea Tyczynska
7930a53a58 Merge pull request #3935 from alphagov/more-detailed-request-go-live-ticket
Make a label more descriptive in request-to-go-live ticket
2021-06-18 10:01:50 +01:00
Pea Tyczynska
d1dc1aec6c Make a label more descriptive in request-to-go-live ticket
We have a label saying "other live services". This label means
other live services for a user making the request, but it could
also be interpreted as other live services for an organisation.

Hence, we are changing the label to "other live services for
that user" to avoid confusion
2021-06-17 17:53:28 +01:00
Ben Thorner
562245d9cd Merge pull request #3931 from alphagov/refactor-email-verify-webauthn
Refactor email revalidation check
2021-06-16 16:24:41 +01:00
Ben Thorner
9177ffa65a Clarify variable name in pagination tests
We should avoid using abbreviations, as they aren't universally
understood i.e. they're not worth the small saving in typing.
2021-06-16 16:08:38 +01:00
Rebecca Law
38edcae68a Merge pull request #3924 from alphagov/letter-permanent-failure
Permanent failure message for letters
2021-06-15 13:43:26 +01:00
Rebecca Law
d7364eb21a Add permanent-failure to the format_notification_status formatter so the status appears on the activity page. 2021-06-15 13:08:07 +01:00
Ben Thorner
1c4a903b65 Move tests to match the module they are in
Previously some of the tests for code in the "formatters" module
were in tests for the "utils" module. This moves them to where
they should be. While two of these methods are probably more utils
than formatters, I'd like to postpone a refactor of that module
for now, and focus on slimming down test for utils/__init__.py.
2021-06-14 12:52:57 +01:00
Ben Thorner
c17d438de8 DRY-up email revalidation check
Previously this was duplicated between the "two_factor" and the
"webauthn" views, and required more test setup. This DRYs up the
check and tests it once, using mocks to simplify the view tests.

As part of DRYing up the check into a util module, I've also moved
the "is_less_than_days_ago" function it uses.
2021-06-14 12:52:54 +01:00
Ben Thorner
edb0b8214d Move pagination utils into own module
This continues the pattern established in [1], just to chip away a
bit more at the random collection of stuff in utils/__init__.py.

[1]: https://github.com/alphagov/notifications-admin/pull/3923
2021-06-14 11:00:05 +01:00
Ben Thorner
44cf2b16b5 Merge pull request #3923 from alphagov/refactor-email-verify
Split out utils code into separate modules
2021-06-14 10:28:28 +01:00
Chris Hill-Scott
2d36ec8214 Add support for the operator channel
Was just in one of those meetings where it felt like writing this would
take less time than I’d already spent talking about its relative
priority…

---

In the admin app you can already set the broadcast channel as 'test', 'severe' or 'government'.

Aim:
- Add the 'operator' channel to the list of channels you can pick for the admin app broadcast services

Note:
- The API already supports this - https://github.com/alphagov/notifications-api/pull/3262
- The CBC proxy does not yet support the operator channel and this will need a separate card. That card has not yet been written because the interface has not been agreed between us and the MNOs yet.
- Will need to have the ability to select the operator channel for just a single MNO like we do for the other channels
- If we add this, we shouldn't actually start using it until the MNO in question gives us the go ahead.

---

https://www.pivotaltracker.com/story/show/178485177
2021-06-14 08:48:12 +01:00
Katie Smith
cc1a8254df Merge pull request #3928 from alphagov/billing-report
Add total_letters to the billing report
2021-06-11 16:43:29 +01:00
Katie Smith
266f1728e4 Merge pull request #3919 from alphagov/refactor-broadcast-settings
Refactor broadcast settings forms
2021-06-11 16:43:09 +01:00
Katie Smith
7498930bf5 Merge pull request #3925 from alphagov/move-init-to-models
Move broadcast model code into an explicit module
2021-06-11 16:42:54 +01:00
Katie Smith
ca84b37179 Merge pull request #3916 from alphagov/consolidate-test
Revise error tests for setting broadcast types
2021-06-11 16:42:38 +01:00
Katie Smith
c1bfc280b7 Add total_letters to the billing report
This adds an extra column to the report that can be downloaded from
`platform-admin/reports/usage-for-all-services`.
2021-06-11 11:08:33 +01:00
Rebecca Law
aedf875110 Improve permenant failure message.
Update delivery status document page.
2021-06-10 16:34:00 +01:00
Ben Thorner
fba8d09875 Move broadcast model code into an explicit module
Previously this was hidden away in an anonymous __init__.py file.
I did think about splitting the models into individual files, like
we do with the top-level models for the app. Since the models are
only imported in one place - i.e. are all used together - it didn't
seem worth the hassle, so I've kept them in one file.
2021-06-10 15:05:38 +01:00
Chris Hill-Scott
e7713de4a5 Merge pull request #3920 from alphagov/refactor-webauthn-model
Refactor User.webauthn_credentials into a ModelList
2021-06-10 11:07:21 +01:00
Rebecca Law
498092f9ac When a letters has passed our validation but is not the postal provider
is unable to print the letter we need to mark the letter as failed.
If we mark the letter as a technical-failure, we say that we will fix
the issue, which is wrong because we can not fix the issue.
If we mark the letter as validation-failed, the letter is in wrong
bucket so the letter is not viewable/downloadable by the client.

This PR updates the message for a letter marked as permanent-failure to
better reflect what has actually happened.
2021-06-10 08:56:14 +01:00
Ben Thorner
adc49b8792 Add __init__.py file to make pytest happy
Otherwise we get the following error:

    ________________________________________ ERROR collecting tests/app/utils/test_user.py ________________________________________
    import file mismatch:
    imported module 'test_user' has this __file__ attribute:
      /Users/benthorner/Documents/Projects/admin/tests/app/models/test_user.py
    which is not the same as the test file we want to collect:
      /Users/benthorner/Documents/Projects/admin/tests/app/utils/test_user.py
    HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
2021-06-09 15:56:34 +01:00
Ben Thorner
35301df908 Relocate unit tests for user permission util
Previously these were lumped together with integration-level tests
for specific endpoints, which test the decorator was applied to the
endpoint in question.
2021-06-09 15:38:28 +01:00
Chris Hill-Scott
f6aa5bdfb8 Refactor User.webauthn_credentials into a ModelList
This saves a bit of repetition, and lets us attach other methods to the
collection, rather than having multiple methods on the user object
prefixed with the same name, or random functions floating about.
2021-06-09 15:21:41 +01:00
Ben Thorner
aafb7e9182 Merge separate test for CSV errors function
It's not clear why this was separate from the other utils tests, or
why it was put under main/ - the code under test wasn't in there.
2021-06-09 15:19:01 +01:00
Ben Thorner
0326005aeb Extract template / csv utility code into modules
This follows a similar approach to the previous commits, noting that
one module depends on the other, so we have to extract both together.
2021-06-09 15:19:00 +01:00
Ben Thorner
2a4aa8b4e1 Extract letter utility code into own module
This provides more room for expansion, and reduces the amount of
arbitrary code in the __init__.py file for the new package.
2021-06-09 13:59:06 +01:00
Ben Thorner
7c27646d6a Extract user utility code into own module
This provides more room for expansion, and reduces the amount of
arbitrary code in the __init__.py file for the new package.
2021-06-09 13:19:05 +01:00
Chris Hill-Scott
eca3454a39 Merge pull request #3914 from alphagov/prune-email-domains-list
Prune the email domains list
2021-06-09 10:25:27 +01:00
Chris Hill-Scott
f8f718dff8 Set user to sign in with newly-added key 2021-06-08 09:31:30 +01:00
Ben Thorner
5bfe5f86de Simplify channel selection using radio buttons
This takes a similar approach as in the previous commit. Since the
"training channel" doesn't really exist, we need some extra code
to pre-select it if a service is already in training mode. As in
the previous commit, I've removed a few non-critical test cases
where we really don't need to test exhaustively.

Note that we also need some specific code to avoid pre-selecting an
option for non-broadcast services, which only used to work by fluke:
we would try to populate the field with (False, None, 'all'), which
isn't a valid combination, so nothing was selected.
2021-06-07 17:51:10 +01:00
Ben Thorner
b38cdcad63 Simplify network choice to optional radio buttons
Previously this field had to mimic the final hyphenated string of
the broadcast account type, even though it was only used to select
one of its components. The new, shorter choices make it easier to
simplify the test for the POST request.

I've also deleted a number of test cases for pre-selected radios.
This functionality isn't critical, so we don't need to exhaustively
test every single possible combination of values.
2021-06-07 17:51:09 +01:00