Commit Graph
100 Commits
Author SHA1 Message Date
Pea Tyczynska e5b1a65f1e Merge pull request #3448 from alphagov/drop_api_key_id_column_from_broadcast_message_table
Drop api_key_id column from broadcast_message table
2022-02-11 14:01:25 +00:00
Pea Tyczynska 66dba8a7c6 Merge pull request #3452 from alphagov/revert-3449-revert-3440-audit-api-key-id-when-cancelling-broadcast-via-api
Audit api key id when cancelling broadcast via api - second try
2022-02-11 12:13:33 +00:00
Pea Tyczynska c55d917b28 Drop api_key_id column from broadcast_message table
This column has been superseded by a new column named
created_by_api_key_id.

Also create constraint checking that we know who created broadcast

Also move data so that constraint is met before instatiating it.
2022-02-11 12:12:00 +00:00
Pea Tyczynska 3dc1907321 Audit api key id when cancelling broadcast via api 2022-02-11 12:01:56 +00:00
Pea Tyczynska 87ab81912c Merge pull request #3453 from alphagov/audit-cancel-alert-via-api-just-first-migration
Migration adding cancelled_by_api_key_id to broadcast_message
2022-02-11 10:34:12 +00:00
Pea Tyczynska 00b63ca5c7 Migration adding cancelled_by_api_key_id to broadcast_message
table. Also add created_by_api_key_id column that will supersede
api_key_id_column.

We do migration and code in separate PRs to make it easier to
downgrade if anything goes wrong.
2022-02-09 17:26:37 +00:00
Pea Tyczynska 9ce6d2fe92 Merge pull request #3449 from alphagov/revert-3440-audit-api-key-id-when-cancelling-broadcast-via-api
Revert "Audit api key id when cancelling broadcast via api"
2022-02-09 11:12:28 +00:00
Pea Tyczynska a780933893 Revert "Audit api key id when cancelling broadcast via api" 2022-02-09 11:01:39 +00:00
Pea Tyczynska d05bff9efc Merge pull request #3440 from alphagov/audit-api-key-id-when-cancelling-broadcast-via-api
Audit api key id when cancelling broadcast via api
2022-02-09 10:15:03 +00:00
Pea Tyczynska f2bef7392e api_key_id column will have to be dropped in a separate PR
This is to avoid situation where we deploy migration already
but don't deploy code yet and they are not compatible for a while
which leads to errors.
2022-02-08 15:57:35 +00:00
Pea Tyczynska 0737eceddb Include check constraint in migration script
Add check constraint that created_by_id should not be null, unless
created_by_api_key_id is not null to the migration script. It is
already in the models file.

Also remove check constraint for cancelled_by_id from models, as
this field would only be filled for broadcasts with cancelled
status.

Also add some spacing in that migration script so it is easier
to read.
2022-02-02 17:21:58 +00:00
Pea Tyczynska ac5967bc5a Merge pull request #3430 from alphagov/rename_billing_report_column_sms_fragments
Rename sms_fragments to sms_chargeable_units
2022-02-02 10:00:30 +00:00
Pea Tyczynska 82f08f230c Save api key id when cancelling broadcast by API call
This is so that we can audit who cancelled the broadcast if
there are any issues.
2022-01-26 17:26:58 +00:00
Pea Tyczynska e1a8219eb1 DB Migration to allow auditing api key id when cancelling broadcast
via API.
2022-01-26 17:26:05 +00:00
Pea Tyczynska 4a90cde701 Merge pull request #3429 from alphagov/cancel_alert_via_api
Cancel broadcast via API
2022-01-21 14:04:35 +00:00
Pea Tyczynska b6dd189462 Test cancel request via API returns 404 if service id does not match 2022-01-20 18:28:10 +00:00
Pea Tyczynska 52dbdb7518 Move validate_and_update_broadcast_message_status to a utils file
This is because that function is used both when broadcast status
is updated via API and via admin, so it's a shared resource.

Also move and update tests for updating broadcast message status
so things are tested at source and repetition is avoided.
2022-01-20 18:14:41 +00:00
Pea Tyczynska c9afb2f038 Remove unnecessary error handling
The context here should be enough for the users, custom error
message is not needed.
2022-01-20 18:14:40 +00:00
Pea Tyczynska c2a389e81a Move updating user validation out of validate_and_update_broadcast_message_status
As only 1 of 2 functions calling it needs that check, it's better
to perform it inside that 1 function.
2022-01-20 18:14:39 +00:00
Pea Tyczynska a4c20e8ba6 Return 404 if reference from cancel message does not match
If the reference from cancel CAP XML we received via API does not
match with any existing broadcast, return 404.

Do the same if service id doesn't match.

Also refactor code to cancel broadcast out into separate function

It should be a separate function that is only called by create_broadcast
function. This will prevent create_broadcast from becoming too
big and complex and doing too many things.
2022-01-19 15:42:27 +00:00
Pea Tyczynska 3b4a9d8942 Cancel broadcast via API
When a service sends us an XML CAP broadcast message with Cancel
status, and that broadcast is in broadcasting state, we cancel it.
2022-01-19 15:42:26 +00:00
Pea Tyczynska 940126abfb Reject unapproved broadcast upon cancel API request
When a service sends us a cancel broadcast XML via API, if that
broadcast was not approved yet, reject it.
2022-01-19 15:41:38 +00:00
Pea Tyczynska d94517d379 Rename sms_fragments to sms_chargeable_units
This field caused some confusion and lots of unnecessary work
    to our colleague because of unclear name.

    The field was named sms_fragments, where in fact the value of
    the field is: those sms fragments that go above free allowance
    multiplied by the rate multiplier.

    The new name was chosen through consultation with colleagues
    who use billing report the most.
2022-01-18 18:03:16 +00:00
Pea Tyczynska 32cd7a0eb6 Merge pull request #3395 from alphagov/fix_org_usage_report
Fix calculating remaining free allowance for SMS
2021-12-21 15:02:54 +00:00
Pea Tyczynska d334e405c5 Refactor tests for sms remainder to make them easier to read 2021-12-21 14:43:56 +00:00
Pea Tyczynska 6c04deaec2 Get rid of unnecessary coalesce 2021-12-14 17:36:03 +00:00
Pea Tyczynska a74d1b026f Fix calculating remaining free allowance for SMS
The way it was done before, the remainder was incorrect in the
billing report and in the org usage query - it was the sms remainder
left at the start of the report period, not at the end of that period.

This became apparent when we tried to show sms_remainder on the org
usage report, where start date is always the start of the financial year.
We saw that sms sent by services did not reduce their free allowance
remainder according to the report. As a result of this, we had to
temporarily remove of sms_remainder column from the report, until
we fix the bug - it has been fixed now, yay!

I think the bug has snuck in partially because our fixtures for testing
this part of the code are quite complex, so it was
harder to see that numbers don't add up. I have added comments
to the tests to try and make it a bit clearer why the results are
as they are.

I also added comments to the code, and renamed some variables,
to make it easier to understand, as there are quite a few
moving parts in it - subqueries and the like.
I also renamed the fetch_sms_free_allowance_remainder method to
fetch_sms_free_allowance_remainder_until_date so it is clearer
what it does.
2021-12-09 18:58:10 +00:00
Pea Tyczynska cf1662e8b1 Merge pull request #3343 from alphagov/publish-alerts
Call publish-govuk-alerts task when alert is sent, cancelled or expires
2021-10-20 14:48:10 +01:00
Pea TyczynskaandKatie Smith 1b6f9505da Call publish-govuk-alerts task when alert expires
The `auto-expire-broadcast-messages` task checks for expired broadcasts
at five minute intervals. This change now calls the
`publish-govuk-alerts` task in govuk-alerts if there are expired
broadcasts so that the site is updated.

Co-authored-by: Katie Smith <katie.smith@digital.cabinet-office.gov.uk>
2021-10-18 08:41:25 +01:00
Pea Tyczynska 9334204cd2 Merge pull request #3305 from alphagov/fetch-alerts-list-for-govuk-alerts
get_broadcasts returns a list of alerts for gov.uk/alerts
2021-08-12 14:33:46 +01:00
Pea Tyczynska 9d2f8347b2 get_broadcasts returns a list of broadcasts for gov.uk/alerts 2021-08-12 14:03:33 +01:00
Pea Tyczynska 0f7f219a55 dao_get_all_broadcast_messages returns just fields govuk alerts need 2021-08-11 14:43:27 +01:00
Pea Tyczynska 74c9ca2bf6 Fetch all broadcast messages that are or were transmitted
Regardless of channel.
Do not include:
- broadcasts older than 25.05.2021
- stubbed broadcasts
- broadcasts that were not transmitted. So only broadcasting,
cancelled and completed make the list;
2021-08-11 14:43:27 +01:00
Pea Tyczynska 82e7724c56 Merge pull request #3296 from alphagov/separate_error_for_cancelling_cancelled_letters
Show separate error for when user tries to cancel letter
2021-08-03 10:29:47 +01:00
Pea Tyczynska 0c8dd247f9 Show separate error for when user tries to cancel letter
that is already cancelled vs when it is too late to
cancel letter vs when we don't know what's the cause
of failure.

This is so we could show useful error messages to the users
and also for better debugging.
2021-07-29 11:32:49 +01:00
Pea Tyczynska 2b2c240bee Update utils version to bring in too_late_to_cancel_letter
We need that method to show right errors to the user
when cancelling letter fails

Update dependencies
2021-07-28 16:33:01 +01:00
Pea Tyczynska 719ba698e2 Merge pull request #3288 from alphagov/bump-utils-to-revert-changes
Bump utils to revert changes to placeholders that introduced a bug
2021-07-09 14:58:02 +01:00
Pea Tyczynska e82b8bc33c Bump utils to revert changes to placeholders that introduced
a bug.
2021-07-09 14:45:41 +01:00
Pea Tyczynska 96f34bbd45 Merge pull request #3286 from alphagov/bump-utils-fix-placeholder-bug
Bump utils to bring in fix for optional placeholder bug
2021-07-09 11:24:42 +01:00
Pea Tyczynska c28e9451d4 Bump moto version to try solve dependencies version conflict
Also update mock import statements in some test files as they
stopped working with this dependency update.
2021-07-08 15:37:19 +01:00
Pea Tyczynska 9e8682ac29 Bump utils to bring in fix for optional placeholder bug
See https://github.com/alphagov/notifications-utils/pull/878 for
details.

Changes we had to make for our app and tests to work correctly
after the dependency updates:

1. Update emergency alerts polygons test because we changed
how exact we are with locations of the points on the map.

2. Use Flask's g object to set additional request attributes

So far we have been storing them in _request_ctx_stack which is
an innard for Flask's request context.

Because of major update to Werkzeug dependency, which Flask relies
on, the way we were using it stopped working, so we had a new
way to set those values.
The way we set those values now, by using g object, seems to also
be favoured in Flask documentation:
https://flask.palletsprojects.com/en/1.1.x/reqcontext/#how-the-context-works
2021-07-08 12:18:09 +01:00
Pea Tyczynska d16ceaf3dc Merge pull request #3245 from alphagov/validate-constraint
Validate ck_user_has_mobile_or_other_auth constraint
2021-05-14 10:12:50 +01:00
Pea Tyczynska 1f6e225a1b Validate ck_user_has_mobile_or_other_auth constraint
This is second step out of two step migration.
We divided it like this to avoid potentially locking
production database for extended amounts of time.
2021-05-13 17:34:35 +01:00
Pea Tyczynska 4b8c389d56 Merge pull request #3243 from alphagov/add-2fa-type-webauthn
Add webauthn as an auth type
2021-05-13 14:31:53 +01:00
Pea Tyczynska d6c3b5e0c9 Do not validate constraint when creating it
To avoid locking production database for extended amounts of time.
2021-05-13 14:13:54 +01:00
Pea Tyczynska 251107029a Add webauthn to tests that include other auth types 2021-05-13 12:44:36 +01:00
Pea Tyczynska 098c6f031b Add webauthn as an auth type.
Both in our models and as a migration to add it to auth_types
table.

Make sure that if we downgrade, we first clean up the data.
2021-05-13 12:44:36 +01:00
Pea Tyczynska d6fead7c04 On update, check that webauthn credential belongs to user 2021-05-12 17:48:38 +01:00
Pea Tyczynska e6291187ba Remove registration_response from webauthn serialize - not needed in admin app
Also fix tests:

First add init file so the tests are found correctly, then update
the tests after we stopped serialising webauthn
registration_response.
2021-05-12 17:48:37 +01:00
Pea Tyczynska 3798a3bd1d Add webauthn_credential table
This is to store data for registered webauthn credentials, so
platform admins can later use them to log in.
2021-05-12 17:48:36 +01:00
Pea Tyczynska df19a91b7f Update error messages after SQLAlchemy version update 2021-04-29 13:32:36 +01:00
Pea Tyczynska 317ae5dbd4 Apply requirements update 2021-04-29 13:32:36 +01:00
Pea Tyczynska e1a626855d Update error messages after SQLAlchemy version update 2021-04-23 16:48:36 +01:00
Pea Tyczynska 66127f3800 Apply requirements update 2021-04-22 16:59:15 +01:00
Pea Tyczynska 38af26cc78 Merge pull request #3199 from alphagov/admin-cancel-broadcast
Allow platform admins to cancel broadcasts.
2021-04-20 14:47:45 +01:00
Pea Tyczynska fce6a2d8dc Allow platform admins to cancel broadcasts.
Also update error message for when someone does not have permissions.

The message referenced approving broadcasts specifically, whereas
people would also see it if they try to cancel or reject
broadcast without permission.

Why we allow platform admins to cancel broadcasts:
we do this so they can react quickly if a broadcast was
approved by accident.
2021-04-20 12:27:38 +01:00
Pea Tyczynska 1a6ad92490 Merge pull request #3194 from alphagov/allow-double-hyphens-in-email-domains
Bump utils to allow double hyphens in email address domain
2021-03-30 11:06:08 +01:00
Pea Tyczynska a3aad24fe1 Bump utils to allow double hyphens in email address domain
It was requested by our user and it is an allowed domain format
with Amazon SES, so we started allowing it in our validation.
2021-03-29 17:53:29 +01:00
Pea Tyczynska 1903e8f268 Merge pull request #3168 from alphagov/client-reference
If client reference not given, try to get it from personalisation
2021-03-24 15:34:34 +00:00
Pea Tyczynska 52c529ab3a Use personalisation to set client_reference for letters
which were sent through Notify interface only. This is done
to avoid performance dip from additional operation for
other notification types.
2021-03-24 14:55:10 +00:00
Pea Tyczynska a2da8bc070 If client reference not given, try to get it from personalisation.
This is mostly useful for letters.

For templated letters sent via interface, whether one-offs
or CSV uploads, we do not give our users a way to set client reference.

Still, they often have a placeholder with reference that we could use
to set client_reference field.

Why is this helpful?

When letter is returned, or when we experience some printing issues,
often it is difficult to identify letters after the retention period.

This change will make it easier for some users to identify letters.
It will have more impact if we inform our users of this in template
editing guidance.
2021-03-24 14:55:10 +00:00
Pea Tyczynska 7c304f6753 Merge pull request #3163 from alphagov/billing-report
Update usage endpoint with billing details for orgs and services
2021-03-23 11:26:23 +00:00
Pea Tyczynska 0dbe4b27c8 Rearrange fixture for readability 2021-03-19 16:50:01 +00:00
Pea Tyczynska 04525dc8c1 Billing report only has services with bills to pay 2021-03-19 16:50:01 +00:00
Pea Tyczynska dd278a0567 Rename get_usage_for_all_services to get_data_for_billing_report
That describes what it does more correctly. As it does not include
services with no usage.
2021-03-19 16:50:01 +00:00
Pea Tyczynska 100d47f4e8 Refactor test and fixture for getting billing report data
Names of services and orgs were confusing, and variable setting
was done in a way that made it easy to introduce errors.

Now hopefully it is more readable and more error-proof.
2021-03-19 16:50:00 +00:00
Pea Tyczynska 4c3d70fd55 Update usage endpoint with billing details for orgs and services 2021-03-19 16:49:48 +00:00
Pea Tyczynska 516a982592 Merge pull request #3159 from alphagov/add-date-to-dvla-email-personalisation
Add date to personalisation for DVLA email
2021-02-24 14:54:06 +00:00
Pea Tyczynska f3e0cfc727 Pull DVLA address from credentials on staging
So that we can test this flow on staging.
2021-02-24 11:34:29 +00:00
Pea Tyczynska 8e3ef5ff05 Add DVLA_EMAIL_ADDRESS to manifest so it gets picked up from
credentials.
2021-02-24 10:32:20 +00:00
Pea Tyczynska 4fc3af9811 Add date to personalisation for DVLA email
Personalisation was missing date attribute. The email still got sent
tonight, just it didn't have a value for date placeholder.
2021-02-24 10:22:22 +00:00
Pea Tyczynska 8e5f956009 Merge pull request #3151 from alphagov/daily-volumes-dvla
Send daily email with volumes of letters and sheets to DVLA
2021-02-23 15:24:31 +00:00
Pea Tyczynska b05ddd4978 Don't donwgrade if the environment is production
Because we are nervous about deleting data on prod, since it can
block the db.
2021-02-23 15:13:53 +00:00
Pea Tyczynska 5c22c926b0 Stub DVLA email for all envs except prod
In prod we will get it from Credentials. In other envs, we don't
really want to send real email.
2021-02-23 15:13:52 +00:00
Pea Tyczynska e0c73ac342 Send daily email with letter and sheet volumes to DVLA 2021-02-23 15:13:19 +00:00
Pea Tyczynska 6dab63130d Make import order alphabetical 2021-02-23 15:13:19 +00:00
Pea Tyczynska c8ffebcce8 Query to get letter and sheet volumes
So we can send daily email with these volumes to DVLA.
2021-02-23 15:13:18 +00:00
Pea Tyczynska 88233543c3 Add email template for daily letter volumes 2021-02-23 15:13:18 +00:00
Pea Tyczynska df00e16100 Merge pull request #3127 from alphagov/stubbed-not-nullable
Make stubbed column on broadcast_message non-nullable
2021-02-10 14:02:17 +00:00
Pea Tyczynska 6e1c5a70c0 Make stubbed column on broadcast_message non-nullable
But first set any null values there to False.
2021-02-09 16:33:11 +00:00
Pea Tyczynska c2191d3a0e Merge pull request #3129 from alphagov/v2-set-stubbed
Set broadcast message to stubbed when posting broadcast via API
2021-02-09 15:37:01 +00:00
Pea Tyczynska 3037bf5fff Set broadcast message to stubbed when posting broadcast via API 2021-02-09 10:41:36 +00:00
Pea Tyczynska e0ddb5a39e Merge pull request #3126 from alphagov/fix-cryptography-build-problem
Pin cryptography to a version < 3.4
2021-02-08 17:25:44 +00:00
Pea Tyczynska 7cc8371c7f Pin cryptography to a version < 3.4
One of our dependencies has a dependency on cryptography, which has
recently released version 3.4.

This version introduced a circular import error
(pyca/cryptography#5756) which was fixed in
3.4.1.

However, 3.4.1 has a different error where it fails because it cannot
find a rust compiler.

The suggested
solutions are:

Install a newer version of pip which will install a pre-compiled
cryptography wheel OR
Have rust installed and available on our PATH so that it can be used
to build the package.
Since we can't change the buildpack's pip version and we cannot install
rust ourselves, the only we're left with is to avoid upgrading to 3.4 -
at least until PaaS updates their python buildpacks.
2021-02-08 17:05:46 +00:00
Pea Tyczynska f8b4c9151c Merge pull request #3122 from alphagov/add-billing-details-orgs
Add billing details for organisation
2021-02-08 16:43:08 +00:00
Pea Tyczynska bbc8cffb5b No need to alter the columns in services, they are already the right type 2021-02-08 10:45:28 +00:00
Pea Tyczynska aa7bc3d9b4 Serialise org notes and billing details 2021-02-05 14:44:43 +00:00
Pea Tyczynska 02bc87c096 Add billing details and notes for organisation table 2021-02-05 14:44:42 +00:00
Pea Tyczynska df4ba22912 Merge pull request #3114 from alphagov/preview-wont-stub
Don't stub broadcasts on preview
2021-01-29 16:00:11 +00:00
Pea Tyczynska 552e543bc2 Don't stub broadcasts on preview
So that MNOs can use training mode accounts to test end-to-end
broadcast sending. This will enable them to approve their own
broadcasts.
2021-01-29 15:49:50 +00:00
Pea Tyczynska 51c0ece130 Merge pull request #3108 from alphagov/stub-training-broadcasts
Stub training broadcasts
2021-01-28 11:58:47 +00:00
Pea Tyczynska d4cc250510 Don't create broadcast provider messages for stubbed broadcasts 2021-01-27 10:20:44 +00:00
Pea Tyczynska 26d6b4a958 Mark broadcast message as stubbed when sent from training account 2021-01-27 10:20:43 +00:00
Pea Tyczynska a93a35de8d Add 'stubbed' column to broadcast_message table
This is a boolean column. It will be set to True for broadcasts
created from training broadcast accounts.

This will help us debug, for example by excluding all the stubbed
broadcasts when we have some trouble with real broadcasts.
2021-01-27 10:20:43 +00:00
Pea Tyczynska dfbd31cef8 Merge pull request #3106 from alphagov/billing-fields-for-service
Add billing details fields to Service model and db table
2021-01-26 15:14:05 +00:00
Pea Tyczynska b3abdfb401 Rename billing contact email and name fields to plural
So:

'billing_contact_email_address' becomes 'billing_contact_email_addresses'
AND
'billing_contact_name' becomes 'billing_contact_names'

This is to signify that each of those fields can contain numerous
items
2021-01-25 17:53:27 +00:00
Pea Tyczynska ffac16a2a0 Add new billing details to test_get_service_by_id 2021-01-25 17:42:18 +00:00
Pea Tyczynska e703d1a172 Add billing details fields to Service model and db table
The fields are:
Purchase order number - string field
Billing contact name - text field to acommodate possible multiple
contacts
Billing contact email address - text field to acommodate possible
multiple contacts
Billing reference - string field

All these fields are nullable. Notify platform admins will be
able to check and edit those values in Service Settings
section in Notify interface.

This will help make billing tasks and reports simpler.

Similar fields will also be added to Organisation model and
db table.
2021-01-20 18:00:43 +00:00
Pea Tyczynska 882da84182 Merge pull request #3096 from alphagov/add-notes-to-service
Add notes column to services table
2021-01-19 14:45:42 +00:00
Pea M. Tyczynska 2749a707f2 Merge pull request #3067 from alphagov/fix-cancel-broadcast
Fix cancel broadcast by converting reference date to string
2020-12-21 13:56:19 +00:00