Removed all existing statsd logging and replaced with:
- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables
- statsd client changed to prefix all stats with "notification.api."
- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way
notifications.api.POST.notifications.send_notification.200
This should allow us to aggregate to the level of
- API or ADMIN
- POST or GET etc
- modules
- methods
- status codes
Finally we count the callbacks received from 3rd parties to mapped status.
"with personalisation" should only be used by the public notification api
"with template" should be used when we want template name, etc details.
also added an xfail test for correctly constructing notification
personalisation
rename the notification_status_schema to make it apparent that it
involves the template, and then don't use it on the job page - the
job page doesn't do anything with the data. won't somebody think of
the cpu cycles! (also means it ignores problems with template
versions)
if api_key used to access endpoint is type team, endpoints only return that type -
will 404 if you provide a different ID. Same applies for normal (normal api keys
cannot see team notifications)
also, for convenience, set sample_notification to supply key_type of KEY_TYPE_NORMAL
by default
moved from notifications/rest -> service/rest and job/rest respectively
endpoint routes not affected
removed requires_admin decorator - that should be set by nginx config
as opposed to python code
notifications, when retrieved by notification id, or service id (i.e.
all notifications for service).
There is a new element returned at top level of notification json called
body, which is the template content merged with personalisation. This
is consistent with api to endpoint to create notification which returns
what was sent as 'body' in json response.
Merging of template with personalisation is done in the
NotificationStatusSchema.
Personalisation data in encrypted before storing in db.
or param errors to raise invalid data exception. That will cause
those responses to be handled in by errors.py, which will log
the errors.
Set most of schemas to strict mode so that marshmallow will raise
exception rather than checking for errors in return tuple from load.
Added handler to errors.py for marshmallow validation errors.
- if a service is in research mode the don't send the notifications via the providers (MMG/SES/etc)
- instead set up a task to mimic those services callbacks - this completes the loop, and show stats, delivery receipts and so on.
- Use the "to" field to choose the response, allows users to create successful and errored notifications
temp fail sms, uses "07833333333"
perm fail sms, uses = "07822222222"
success = "07811111111" (or anything else)
success email = "delivered@simulator.notify"
perm fail = "perm-fail@simulator.notify"
temp fail = "temp-fail@simulator.notify"
If the notification has a status == sending then update the status otherwise do not update the status.
In other words do not change the status more than once.
- new client for statsd, follows conventions used elsewhere for configuration
- client wraps underlying library so we can use a config property to send/not send statsd
Added statsd metrics for:
- count of API successful calls SMS/Email
- count of successful task execution for SMS/Email
- count of errors from Client libraries
- timing of API calls to third party clients
- timing of how long messages live on the SQS queue
Update notifications/sms|email endpoint to send the template version to the queue.
Update the process_job celery talk to send the template version to the queue.
When the send_sms|send_email task runs it will get the template by id and version.
Created a data migration script to add the template_vesion column for jobs and notifications.
The existing jobs and notifications are given the template_version of the current template.
There is a chance this is the wrong template version, but deemed okay since the application is not live.
Create unit test for the dao_get_template_versions method.
Rename /template/<id>/version to /template/<id>/versions which returns all versions for that template id and service id.
It’s nicer to have emails with a sender name, as well as the raw email
address.
Amazon SES can acheive this by using the format
```
"Sender name" <sender.name@domain.com>
```
— http://docs.aws.amazon.com/ses/latest/DeveloperGuide/email-format.html
We also have to remove all non-ASCII characters from the sender name,
because SMTP only supports 7-bit ASCII:
> A field name MUST be composed of printable US-ASCII characters (i.e.,
> characters that have values between 33 and 126, inclusive), except
> colon.
— http://www.ietf.org/rfc/rfc5322.txt
We use the service name as the sender name when:
- sending emails from the API
- sending emails from a CSV file
We use GOV.UK Notify as the sender name when:
- sending invitation emails
- sending password reset emails
Refactor process_firetext_responses
Removed the abstract ClientResponses for firetext and mmg. There is a map for each response to handle the status codes sent by each client.
Since MMG has about 20 different status code, none of which seem to be a pending state (unlike firetext that has 3 status one for pending - network delay).
For MMG status codes, look for 00 as successful, everything else is assumed to be a failure.
