We want to log the usage of the various API clients we have so that we understand when they can be cycled.
To this end we are going to count usage in statsd.
All notify clients have a suer agent, of the format: NOTIFY-API-{LANGUAGE}-CLIENT/version.number
For example, NOTIFY-API-PYTHON-CLIENT/3.0.0
We convert that into a statsd/graphite friendly string of the format: notify-api-python-client.3-0-0
So we can subdivide on client and client version on our dashboards.
Present but unknown User agents are records as "non-notify-user-agent"
Missing are presented as "unknown"
We're formally using the ISO 8601 UTC datetime format, and so the
correct way to output the data is by appending the timezone.
("Z" in the case of UTC*).
Unfortunately, Python's `datetime` formatting will just ignore the
timezone part of the string on output, which means we just have to
append the string "Z" to the end of all datetime strings we output.
Should be fine, as we will only ever output UTC timestamps anyway.
* https://en.wikipedia.org/wiki/ISO_8601#UTC
The new 'v2' API wants to return less data than the previous one,
which was sending back tons of fields the clients never used.
This new route returns only useful information, with the JSON
response dict being built up in the model's `.serialize()` method.
Note that writing the test for this was a bit painful because of
having to treat loads of keys differently. Hopefully we think this
is a good way to write this test, because if we don't, we should
start thinking of a better way to check the values are what we
expect.
- Use these validation methods in post_sms_notification and the version 1 of post_notification.
- Create a v2 error handlers.
- InvalidRequest has a to_dict method for private and v1 error responses and a to_dict_v2 method to create the v2 of the error responses.
- Each validation method has extensive unit tests, so the unit test for the endpoint do not need to check every error case, but check that the error handle formats the message correctly.
- The format of the error messages is still a work on progress.
- This version of the api could be deployed without causing a problem to the application.
- The new endpoing is still a work in progress and is not being used yet.
- this allows us to send a notification to a provider by means of an API call
- This is in addition to the celery code.
- idea is that we can use this method to help speed up throughput by generating API traffic by node/lambda etc to supplement the celery code in times of high load.
it was causing a bug where a local variable service was not being
instantiated and we were trying to operate on the blueprint instead
it's being used in so few places it makes sense to rename it
Removed all existing statsd logging and replaced with:
- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables
- statsd client changed to prefix all stats with "notification.api."
- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way
notifications.api.POST.notifications.send_notification.200
This should allow us to aggregate to the level of
- API or ADMIN
- POST or GET etc
- modules
- methods
- status codes
Finally we count the callbacks received from 3rd parties to mapped status.
Adds a new endpoint (`/spec`) which returns a the specification of the
API in Swagger-formatted JSON.
This means we will have something to point frontends at, so we can
evaluate which ones we like.
Right now it’s all hand-defined. If we were consistent about our use of
Marshmallow we could generated the spec from the Marshmallow schemas.
- new client for statsd, follows conventions used elsewhere for configuration
- client wraps underlying library so we can use a config property to send/not send statsd
Added statsd metrics for:
- count of API successful calls SMS/Email
- count of successful task execution for SMS/Email
- count of errors from Client libraries
- timing of API calls to third party clients
- timing of how long messages live on the SQS queue
Refactor process_firetext_responses
Removed the abstract ClientResponses for firetext and mmg. There is a map for each response to handle the status codes sent by each client.
Since MMG has about 20 different status code, none of which seem to be a pending state (unlike firetext that has 3 status one for pending - network delay).
For MMG status codes, look for 00 as successful, everything else is assumed to be a failure.
This PR removes the need for the email_safe function. The api does not create the email_from field for the service.
Tests were updated to reflect this change.
