Commit Graph

8262 Commits

Author SHA1 Message Date
David McDonald
54fe8ee68d Remove old todo for support of draft to broadcasting transition
It looks like we were allowing broadcasts to transition from draft to
broadcasting in one go. This isn't valid now. It should go draft,
pending approval and then broadcasting.

It looks like this was a leftover bit of support in our code for when we
were building stuff out and is no longer needed.
2021-06-15 17:18:54 +01:00
Rebecca Law
6d09313b18 Merge pull request #3268 from alphagov/update-job-sooner
Update the job_status to in-progress sooner.
2021-06-15 15:45:52 +01:00
Ben Thorner
92d52de68f Merge pull request #3254 from alphagov/fix-ci-bootstrap
Fix 'flask db upgrade' not working on CI
2021-06-15 10:23:33 +01:00
Ben Thorner
c89200a833 Fix 'flask db upgrade' not working on CI
Related to: https://github.com/alphagov/notifications-aws/pull/905

Previously this would fail because the Docker image we use for CI
builds doesn't have an 'environment.sh' file; it uses preset env
vars instead. This makes the command to upgrade the DB optional -
if the env file is missing, the error should be self evident.
2021-06-15 09:58:19 +01:00
Rebecca Law
8af10eb1f0 Update the job_status to in-progress sooner.
We had a situation where the delivery-worker app instance was terminated before the job was marked as `in-progress`, presumably because the query to check the daily limits was taking too long to complete.
If the job was in progress the `check_job_status` task would have restarted the job.
Updating the status to in-progress sooner will help.
2021-06-15 07:58:17 +01:00
Katie Smith
ec646c3071 Merge pull request #3264 from alphagov/billing-report
Add new total_letters field to the billing report data
2021-06-11 14:45:03 +01:00
Katie Smith
0148b3dba6 Add new total_letters field to the billing report data
This adds total_letters to the data that is returned by the
`/platform-stats/data-for-billing-report` endpoint so that we can add
total letters as a column in the CSV file that can be downloaded.
2021-06-11 11:31:22 +01:00
David McDonald
6a99a1fbc2 Merge pull request #3262 from alphagov/operator-channel
Operator channel
2021-06-10 09:57:33 +01:00
David McDonald
be035664c4 Add operator channel to broadcast settings route
Looks identical to the government channel in terms of the interface
2021-06-09 13:49:06 +01:00
David McDonald
d18bf2c48a Add operator channel migration
Looks identical to the government channel migration 354
2021-06-09 13:48:02 +01:00
Leo Hemsted
8e1a144f87 Merge pull request #3229 from alphagov/data-error
make sure all non-uuid service ids 403 in api keys
2021-06-07 14:09:22 +01:00
Leo Hemsted
c53ed4107b Merge pull request #3260 from alphagov/verify-to-complete
rename verify webauth endpoint to complete
2021-06-04 13:25:10 +01:00
Leo Hemsted
542b151875 rename verify webauth endpoint to complete
it doesn't really do any verification - that's the webauthn code in the
browser and the admin app that does that. Instead, this completes the
login flow, by marking the user as logged in in the database. Added a
docstring that explains this process a bit more, and also added a new
route: /<id>/complete/webauthn. We'll move the admin code over to use
this new url in time
2021-06-03 17:12:19 +01:00
Rebecca Law
138bef9493 Merge pull request #3259 from alphagov/revert-3256-no-totals-for-high-volume-services
Revert "Do not include today's totals"
2021-06-02 16:14:57 +01:00
Rebecca Law
684a882cf3 Revert "Do not include today's totals" 2021-06-02 16:06:33 +01:00
Rebecca Law
c668bed9d3 Merge pull request #3256 from alphagov/no-totals-for-high-volume-services
Do not include today's totals
2021-06-02 15:08:45 +01:00
Rebecca Law
a341536de0 - Add comment to test and new if statement
- Update assert in test
2021-06-02 14:13:31 +01:00
Rebecca Law
0e0c911517 Merge pull request #3252 from alphagov/upgrade-warning-to-error
Add a error log for alert tasks.
2021-06-02 13:48:01 +01:00
Rebecca Law
b170b5ed80 This change is a temporary fix to allow users for high volume services to use the admin app.
The trouble is the aggregate query to return the big blue numbers on the dashboard and /notifications/{notification_type} page is taking too long to return.
I have some ideas on how to improve the query, but should take some time to do some more research and test. In the meantime, let's just ignore "todays" total numbers for the high volume services. There are only two services that this will affect.
2021-06-02 10:31:38 +01:00
Rebecca Law
3159334907 Merge pull request #3249 from alphagov/update-stats-query
Update the dao_fetch_todays_stats_for_service query.
2021-06-02 09:18:19 +01:00
Rebecca Law
d4b5373ee5 Update tests for BST to be on the day BST starts.
Add a test for just after midnight of the date the stats are collected.
2021-06-01 16:07:37 +01:00
David McDonald
8b19719b48 Merge pull request #3255 from alphagov/revert-3250-bump-utils-new-invaild-address-char
Revert "Bump utils version for new invalid address character"
2021-06-01 13:06:18 +01:00
David McDonald
04e23ca6a9 Revert "Bump utils version for new invalid address character" 2021-06-01 10:53:28 +01:00
Rebecca Law
4f40c12f6e Merge pull request #3250 from alphagov/bump-utils-new-invaild-address-char
Bump utils version for new invalid address character
2021-05-27 14:22:10 +01:00
Rebecca Law
50de85988e Fix dependency issues
We haven't bumped the test version for a while.
Also bumped the version of Flask and itsdangerous.
In order to fix flask warnings I needed to changed how the blueprints were registerd.
2021-05-27 13:02:24 +01:00
Rebecca Law
ed5e3b3d9c Removed the end date in the filter.
It's always going to be in the future anyway.
After some analysis the query does perform better without it.
I'll make a note to update other queries where we get todays
notification data to remove the end date filter in a separate PR.
2021-05-26 13:47:53 +01:00
Rebecca Law
dfacba5053 Add a test for a bst date 2021-05-26 11:50:05 +01:00
Rebecca Law
1bf5ce08b2 Add a error log for alert tasks.
Many of the team members do not look at emails from zendesk, adding a current_app.logger.error message for things we care about to give developers a better chance of seeing them.
I have purposely not added an erro log for `check_for_services_with_high_failure_rates_or_sending_to_tv_numbers` because it's not something we need to look at immediately.
2021-05-26 11:06:21 +01:00
Rebecca Law
f66f0a2e2d Bump moto dependency to resolve dependency conflicts 2021-05-25 14:20:25 +01:00
Rebecca Law
bd1498f49f Bump utils version which contains ~ as an invalid first character for a
postal address.
2021-05-25 08:29:25 +01:00
Rebecca Law
782514a0f1 Update the dao_fetch_todays_stats_for_service query.
We have an index on Notifications(service_id, created_at), by updating the query to use between created_at rather than date(created_at) this query will use the index. Changing the query plan to use an index scan rather than a sequence scan, see query plans below.
This query is still rather slow but is improved by this update.

https://www.pivotaltracker.com/story/show/178263480

explain analyze
SELECT notification_type, notification_status, count(id)
FROM notifications
WHERE service_id = 'e791dbd4-09ea-413a-b773-ead8728ddb09'
AND date(created_at) = '2021-05-23'
AND key_type != 'test'
GROUP BY notification_type, notification_status;
                                                                                     QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Finalize GroupAggregate  (cost=6326816.31..6326926.48 rows=24 width=22) (actual time=91666.805..91712.976 rows=10 loops=1)
   Group Key: notification_type, notification_status
   ->  Gather Merge  (cost=6326816.31..6326925.88 rows=48 width=22) (actual time=91666.712..91712.962 rows=30 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Partial GroupAggregate  (cost=6325816.28..6325920.31 rows=24 width=22) (actual time=91662.907..91707.027 rows=10 loops=3)
               Group Key: notification_type, notification_status
               ->  Sort  (cost=6325816.28..6325842.23 rows=10379 width=30) (actual time=91635.890..91676.225 rows=270884 loops=3)
                     Sort Key: notification_type, notification_status
                     Sort Method: external merge  Disk: 10584kB
                     Worker 0:  Sort Method: external merge  Disk: 10648kB
                     Worker 1:  Sort Method: external merge  Disk: 10696kB
                     ->  Parallel Seq Scan on notifications  (cost=0.00..6325123.93 rows=10379 width=30) (actual time=0.036..91513.985 rows=270884 loops=3)
                           Filter: (((key_type)::text <> 'test'::text) AND (service_id = 'e791dbd4-09ea-413a-b773-ead8728ddb09'::uuid) AND (date(created_at) = '2021-05-23'::date))
                           Rows Removed by Filter: 16191366
 Planning Time: 0.760 ms
 Execution Time: 91714.500 ms
(17 rows)

explain analyze
SELECT notification_type, notification_status, count(id)
FROM notifications
WHERE service_id = 'e791dbd4-09ea-413a-b773-ead8728ddb09'
AND created_at  >= '2021-05-22 23:00'
and created_at < '2021-05-23 23:00'
AND key_type != 'test'
GROUP BY notification_type, notification_status;
                                                                                                                       QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Finalize GroupAggregate  (cost=2114273.37..2114279.57 rows=24 width=22) (actual time=21032.076..21035.725 rows=10 loops=1)
   Group Key: notification_type, notification_status
   ->  Gather Merge  (cost=2114273.37..2114278.97 rows=48 width=22) (actual time=21032.056..21035.703 rows=30 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Sort  (cost=2113273.35..2113273.41 rows=24 width=22) (actual time=21029.261..21029.265 rows=10 loops=3)
               Sort Key: notification_type, notification_status
               Sort Method: quicksort  Memory: 25kB
               Worker 0:  Sort Method: quicksort  Memory: 25kB
               Worker 1:  Sort Method: quicksort  Memory: 25kB
               ->  Partial HashAggregate  (cost=2113272.56..2113272.80 rows=24 width=22) (actual time=21029.228..21029.230 rows=10 loops=3)
                     Group Key: notification_type, notification_status
                     ->  Parallel Bitmap Heap Scan on notifications  (cost=114455.71..2111695.14 rows=210322 width=30) (actual time=4983.790..20960.581 rows=271217 loops=3)
                           Recheck Cond: ((service_id = 'e791dbd4-09ea-413a-b773-ead8728ddb09'::uuid) AND (created_at >= '2021-05-22 23:00:00'::timestamp without time zone) AND (created_at < '2021-05-23 23:00:00'::timestamp without time zone))
                           Rows Removed by Index Recheck: 1456269
                           Filter: ((key_type)::text <> 'test'::text)
                           Heap Blocks: exact=12330 lossy=123418
                           ->  Bitmap Index Scan on ix_notifications_service_created_at  (cost=0.00..114329.51 rows=543116 width=0) (actual time=4973.139..4973.140 rows=813671 loops=1)
                                 Index Cond: ((service_id = 'e791dbd4-09ea-413a-b773-ead8728ddb09'::uuid) AND (created_at >= '2021-05-22 23:00:00'::timestamp without time zone) AND (created_at < '2021-05-23 23:00:00'::timestamp without time zone))
 Planning Time: 0.191 ms
 Execution Time: 21035.770 ms
(21 rows)
2021-05-25 08:00:24 +01:00
Leo Hemsted
70ff00f2c3 Merge pull request #3247 from alphagov/webauthn-login-endpoint
add endpoint for verifying webauthn login
2021-05-21 12:26:02 +01:00
Leo Hemsted
c1b08e4cbc make sure all non-uuid service ids 403 in api keys
previously 'invalid-strings' would be handled, but integers would just
return 500.
2021-05-19 08:57:31 +01:00
Rebecca Law
7b5eb5f905 Fix import order check 2021-05-19 08:21:35 +01:00
Leo Hemsted
00b0227007 add endpoint for verifying webauthn login
with sms and email auth the api handles verifying logins in the
`/<user_id>/verify/code` endpoint, when it checks the code is valid etc.
The admin app has already done this for webauthn logins, but we still
need an API endpoint so that we can set up the user's db entry to have
a new logged in timestamp, a new session id (this is important for
logging out other browser sessions), etc.

Also, we need to be able to make sure that the user's max login count
isn't exceeded. If it's exceeded, we shouldn't let them log in even with
a valid webauthn check.

This endpoint is a POST where the admin passes in a json dict with key
"succesful" being True or False. True sets up the db stuff as mentioned.
False just increments the failed login count.
2021-05-17 20:37:46 +01:00
Sakis
3702d4eae8 Merge pull request #3246 from alphagov/add-splunk-service
Add splunk service
2021-05-14 18:56:34 +03:00
Pea Tyczynska
d16ceaf3dc Merge pull request #3245 from alphagov/validate-constraint
Validate ck_user_has_mobile_or_other_auth constraint
2021-05-14 10:12:50 +01:00
sakisv
9b34a2a9a2 Add splunk service
This will allow shipping app and router logs to splunk[1]

This will is only bound on the API because we're only interested in
paas router logs for the time being

1: https://github.com/alphagov/paas-csls-splunk-broker/blob/main/docs/user-guide.md
2021-05-14 11:17:26 +03:00
Pea Tyczynska
1f6e225a1b Validate ck_user_has_mobile_or_other_auth constraint
This is second step out of two step migration.
We divided it like this to avoid potentially locking
production database for extended amounts of time.
2021-05-13 17:34:35 +01:00
Pea Tyczynska
4b8c389d56 Merge pull request #3243 from alphagov/add-2fa-type-webauthn
Add webauthn as an auth type
2021-05-13 14:31:53 +01:00
Pea Tyczynska
d6c3b5e0c9 Do not validate constraint when creating it
To avoid locking production database for extended amounts of time.
2021-05-13 14:13:54 +01:00
Pea Tyczynska
251107029a Add webauthn to tests that include other auth types 2021-05-13 12:44:36 +01:00
Pea Tyczynska
098c6f031b Add webauthn as an auth type.
Both in our models and as a migration to add it to auth_types
table.

Make sure that if we downgrade, we first clean up the data.
2021-05-13 12:44:36 +01:00
Leo Hemsted
ef3d6aded4 Merge pull request #3239 from alphagov/add-webauthn
Add webauthn
2021-05-13 10:42:31 +01:00
Leo Hemsted
c190886bfe tweak webauthn rest errors
simplify logic by changing the dao function to require a user id and a
webauthn cred id. Note that this changes the response from a 400 to a
404 if the cred is for a different user than the supplied id.

give a minimum length to the text fields in POSTS to create/update a
credential to avoid surprising unexpected edge cases involving empty
string names etc.
2021-05-12 17:48:38 +01:00
Pea Tyczynska
d6fead7c04 On update, check that webauthn credential belongs to user 2021-05-12 17:48:38 +01:00
Pea Tyczynska
e6291187ba Remove registration_response from webauthn serialize - not needed in admin app
Also fix tests:

First add init file so the tests are found correctly, then update
the tests after we stopped serialising webauthn
registration_response.
2021-05-12 17:48:37 +01:00
Leo Hemsted
e62e050963 add webauthn crud endpoints
added some simple validation to the delete endpoint for sanity, but
generally my assumption is that more validation will happen on the admin
side.

noteably im not checking whether the credentials are duplicated, nor is
there a uniqueness constraint in the database - I'm not sure if the
credential blob will always reliably be equivalent, and I believe the
browser should hopefully take care of dupes.
2021-05-12 17:48:37 +01:00
Leo Hemsted
500feba50d add name/id and consolidate webauthn types in model/table
so we can be in line with what the admin handles, and keep it simple on
the api side and do as little manipulation of binary data as possible.

### Minor changes

* id is a UUID we can use for referencing within notify. No relation to
 the key itself.
* name is a user viewable name that can be set/edited
* fix updated_at to have onupdate, not default

### Simplify the webauthn data

credential_data is the data we store about an authenticator that we'll
use to identify the key when logging in. includes the credential_id, the
public_key, and the aaguid (which identifies the authenticator
make/model)

registration_response is the data containing audit information - in the
future we can use this to ensure that the authenticators used are of
high quality.

both of these fields are CBOR (a kind of binary json), encoded in
base64 so that they can be embedded within our regular JSON api
endpoints. we don't anticipate the api ever needing to interact with
this data directly.
2021-05-12 17:48:37 +01:00
Pea Tyczynska
3798a3bd1d Add webauthn_credential table
This is to store data for registered webauthn credentials, so
platform admins can later use them to log in.
2021-05-12 17:48:36 +01:00