Commit Graph

7813 Commits

Author SHA1 Message Date
Chris Hill-Scott
e161f6e4a1 Require reference if template not provided
In the admin app we need something to use in show in lieu of template
name when a template isn’t used. Let’s store this in the reference field
for now.
2021-01-15 14:57:36 +00:00
Chris Hill-Scott
0510311d63 Don’t require template when content is provided
So that the admin app can create broadcasts without a template it needs
to be allowed to create broadcasts from content instead.
2021-01-15 14:57:36 +00:00
Chris Hill-Scott
01504ed760 Merge pull request #3090 from alphagov/migration-allow-broadcasts-created-by-api
Allow broadcast messages to be created with API key
2021-01-15 14:05:11 +00:00
Chris Hill-Scott
78e87857e3 Don’t serialize nullable UUID columns to 'None'
We should return a proper `None` instead, so it gets JSONified as `null`
and returns what you’d expect when doing `bool(model.field)`
2021-01-15 13:15:00 +00:00
Chris Hill-Scott
fe420448e1 Allow broadcast messages to be created with API key
When we have a public API there will be no human creating the broadcast
message. Instead it will be created by an API integration, authenticated
by a key (just like for emails or texts).

This updates the database to:
- add a new foreign key from BroadcastMessages to API keys
- add a `reference` column

It doesn’t change the model yet, because the model is used by previous
migrations, so would cause them to fail when run before the new columns
exist. We can make this change in later pull requests.
2021-01-15 12:52:24 +00:00
Chris Hill-Scott
8d86d70739 Rewrite previous migration in raw SQL
We shouldn’t import models into migrations because if the model changes
later down the line then the migration can’t be re-run at a later date
(for example to rebuild a database from scratch).

We don’t need to encode the content before storing it (we’ll always do
that before rendering/sending) so we don’t need to use
`BroadcastMessageTemplate`.

And given that no past broadcasts will have personalisation, we don’t
need to replace the personalisation in the template before rendering it.
So we can just copy the raw content from the templates table.
2021-01-15 12:52:24 +00:00
David McDonald
36044ca71f Merge pull request #3094 from alphagov/three-proxy
Three proxy
2021-01-15 09:38:57 +00:00
David McDonald
060ee54a74 Enable Three CBC 2021-01-14 11:52:23 +00:00
David McDonald
ff193387d1 Add proxy client for Three
Three uses the One 2 Many technology so should work in the same way as
our proxy for EE
2021-01-14 11:44:46 +00:00
David McDonald
f3ee2cdd48 Merge pull request #3086 from alphagov/lambda-errors
Failover to second lambda on error
2021-01-14 11:15:34 +00:00
David McDonald
3216a74fbd Don't try failover for canary
No point trying, it's the same lambda. As `_invoke_lambda` currently
takes a bytes payload, rather than a json payload, it meant the decision
between encoding the payload in the canary or moving the encoding into
the `_invoke_lambda` function. We decided to go for the former as the
lesser of two evils. We may end up doing the encoding twice in the case
of a failover but this avoids us having to put the encoding in our code
in several places (for example the canary and also soon to be the link
tests).
2021-01-14 11:00:38 +00:00
Pea Tyczynska
b5a33ded98 Retry with failover lambda for FunctionError and status > 299
For all FunctionErrors, and for invoke errors (status > 299) we
want to retry with failover lambda.

We are doing this, because if there is a connection or other error
with one lambda, the failover lambda may still work and it's
worth trying.

With time, we will probably have more complex retry flow, depending
on the error and even maybe differing for each MNO (broadcast provider).
2021-01-14 10:45:29 +00:00
David McDonald
fb5b05a983 Merge pull request #3089 from alphagov/everyday-2nd-class-alert
Alert on 2nd class letters still in sending everyday
2021-01-13 12:16:55 +00:00
Leo Hemsted
5ade5ba13f Merge pull request #3087 from alphagov/migrate-broadcast
add content to old broadcast messages with no content
2021-01-13 11:39:30 +00:00
David McDonald
c3ef23c771 Alert on 2nd class letters still in sending everyday
In 8285ef5f89
we turned off alerting on 2nd class letters still being in sending on
certain days of the week because we were only sending letters out on
Mon, Wed, Fri.

Now we have swapped back to sending out 2nd class letters on all
workdays so this change can be reverted. Note, I haven't reverted the
commit exactly but more so the behaviour, whilst leaving in some tests
to explicitly test 2nd class letters for the alert in case we change
this again.
2021-01-13 11:21:27 +00:00
Rebecca Law
4529b92e23 Merge pull request #3088 from alphagov/update-org-query
Change the sort order for the organisation usage page
2021-01-13 10:28:57 +00:00
Leo Hemsted
54495b4e14 add content to old broadcast messages with no content
new broadcast messages will have content filled whether they have a
tempalte or not, but old ones won't so populate.

Stole the session constructor from 0044_jos_to_notification_hist.py
2021-01-13 10:09:16 +00:00
Pea Tyczynska
1aff854afd Create logs for invoking and finishing lambda, and for retry.
Those logs will give us extra visibility into lambda invocation
process.
2021-01-12 15:34:48 +00:00
Pea Tyczynska
d7661abe81 Move variable used in tests to top of file for more DRY code 2021-01-12 15:34:47 +00:00
David McDonald
24f52721f3 Retry with second lambda if connection error
Note, we assume whenever there is a `FunctionError` that there will be a
payload that contains an `errorMessage` key. It's implied implicitely in
the docs but it's not very explicit.

https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html#API_Invoke_ResponseSyntax
2021-01-12 15:34:47 +00:00
David McDonald
9da8e54d69 Define failover lambdas
We will need a lambda to failover to if the first lambda fails. This
isn't so much a case of the lambda itself failing, as it is a cross
availability zone resource automatically, it's more in case something in
the networking goes down in our AZ and therefore the lambda can't call
out to the CBC. In this case, we will be able to swap to using the
second AZ by calling the second lambda.
2021-01-12 15:34:47 +00:00
David McDonald
1e537d507b Make lambda_name abstract property
As we require all instances to have it
2021-01-12 15:34:46 +00:00
David McDonald
5a46662c28 Abstract invoking of lambda
This is to prepare us for where when we try and send/cancel a broadcast
we may need to invoke more than one lambda. This might happen if we call
the invoke the first lambda, we get an error and therefore we try and
invoke a failover/second lambda. Then `_invoke_lambda` will be
responsible for the call to AWS whereas `_invoke_lambda_with_failover`
will be responsible more for picking the lambda and deciding on retry
behaviour if failure cases.
2021-01-12 15:34:46 +00:00
Rebecca Law
e05e9bb5e0 Change the sort order for the organisation usage page.
Ensure the archived services are at the bottom of the list. The organisation trial mode page already sorts the archived services to the bottom.
2021-01-12 09:44:35 +00:00
Leo Hemsted
4980c3e0fa Merge pull request #3085 from alphagov/fix-broadcast-migration
Fix broadcast migration
2021-01-11 16:13:52 +00:00
Leo Hemsted
400dfe0217 allow broadcasts to have a template and no content
ensures code remains backwards compatible during the deploy. this commit
should be reverted once all broadcast_message.content fields have been
back-filled.
2021-01-11 15:56:40 +00:00
Leo Hemsted
91abe6d55f allow null content in migration
because existing rows won't have any content populated yet.
2021-01-11 15:56:11 +00:00
Leo Hemsted
a3184c53e9 Merge pull request #3084 from alphagov/broadcast-job-content
add content to broadcast_message and make template fields nullable
2021-01-11 14:44:09 +00:00
Leo Hemsted
2e929754ff add content to broadcast_message and make template fields nullable
we want to be able to create broadcast messages without templates. To
start with, these will come from the API, but in future we may want to
let people create via the admin interface without creating a template
too.

populate a non-nullable content field with the values supplied via the
template (or supplied directly if via api).
2021-01-08 18:58:17 +00:00
Sakis
88a6b7729e Merge pull request #3082 from alphagov/fix-sender-logging
Add disk space check for sender worker
2021-01-06 10:58:30 +02:00
sakisv
9bb9070ba0 Add disk space check for sender worker
Reused the existing `ensure_celery_is_running` function to terminate the
script
2021-01-04 14:01:19 +02:00
Chris Hill-Scott
d55b66a6d8 Merge pull request #3075 from alphagov/cache-provider-lookup
Cache provider lookups for 10 seconds
2021-01-04 10:00:25 +00:00
Leo Hemsted
386c3671bb Merge pull request #3073 from alphagov/pyup-scheduled-update-2020-12-23
Scheduled weekly dependency update for week 51
2020-12-31 14:37:36 +00:00
Leo Hemsted
4814c66c1d fix schema metaclasses
marshmallow v0.22.0 added load_instance and include_relationship
options, which we need to keep old ModelSchema code working
2020-12-31 14:13:05 +00:00
Leo Hemsted
a33ec5c7f1 remove deprecated ModelSchema class 2020-12-31 13:56:20 +00:00
Leo Hemsted
ee2bec2f72 pin marshmallow-sqlalchemy
to keep marshmallow <=3.0 dep
2020-12-31 13:56:18 +00:00
Leo Hemsted
156c7aa32a bump python client
brings in jwt2.0 compat
2020-12-31 13:56:04 +00:00
Leo Hemsted
1da16eda23 freeze reqs 2020-12-31 13:55:37 +00:00
pyup-bot
b298440f00 Update sqlalchemy from 1.3.20 to 1.3.22 2020-12-31 13:55:37 +00:00
pyup-bot
97d35b86b5 Update pyjwt from 1.7.1 to 2.0.0 2020-12-31 13:55:37 +00:00
pyup-bot
659a43e435 Update cachetools from 4.1.1 to 4.2.0 2020-12-31 13:55:37 +00:00
pyup-bot
e4c5633150 Update eventlet from 0.29.1 to 0.30.0 2020-12-31 13:55:37 +00:00
pyup-bot
0c0821b9f9 Update prometheus-client from 0.8.0 to 0.9.0 2020-12-31 13:55:37 +00:00
pyup-bot
39877e1e40 Update marshmallow-sqlalchemy from 0.23.1 to 0.24.1 2020-12-31 13:55:37 +00:00
pyup-bot
e560b4a972 Update flask-marshmallow from 0.11.0 to 0.14.0 2020-12-31 13:55:37 +00:00
pyup-bot
20994c2d5d Update cffi from 1.14.3 to 1.14.4 2020-12-31 13:55:37 +00:00
David McDonald
57f5bd76de Merge pull request #3081 from alphagov/ses-error-logs
SES error logs
2020-12-31 13:13:20 +00:00
Leo Hemsted
d470c928cd Merge pull request #3072 from alphagov/doc-dl-exc
handle doc dl connection errors correctly
2020-12-31 11:24:00 +00:00
David McDonald
56879d0d22 Make sure error message is logged as part of the exception 2020-12-31 11:08:09 +00:00
Chris Hill-Scott
8834377a5d Merge pull request #3074 from alphagov/serialise-process-type
Serialise process_type for template history
2020-12-31 09:54:00 +00:00