We wrote to the queue as a performance optimisation , however dev found it confusing to not have immediate access to the notification as it may be perished some minutes later under periods of load. Additionally we had a couple of DB issues which led to us dropping notifications.
Pushing the DB write to earlier in the flow makes the system a little more robust in the early days, we may want to change this when the traffic increases.
Logic:
- live services don't check days limit for now
- restricted services check limits
(caveat) simulate keys aren't checking day limit even in restricted mode.
When you use a simulate API key it should behave like a live service,
except for actually sending the messages. There should be no limits even
if the service is in trial mode.
This commit removes the restriction on sending messages when you’ve sent
up to your daily limit.
When you use a simulate API key it should behave like a live service,
except for actually sending the messages. There should be no limits even
if the service is in trial mode.
This commit removes the restriction on sending messages to peole outside
your team when you’re in trial mode.
Previously there were 4 queues for sending messages
The was based on the fact that each notification has 2 actions - persist in the database and send to provider.
Two queues supported the CSV upload - for the first of these tasks
- bulk-email
- build-sms
And there were two more queues for the tasks that make the 3rd party client calls.
- sms
- email
API Calls just used the latter two queues for both tasks
Added four new queues
- db-email
- db-sms
- send-sms
- send-email
So an API call puts a notification into the db-[type] queue first, which then puts the notification into the send-[type] queue
Build queues stay as before.
This will allow us to target processing of these tasks with separate workers to manage these differently.
- As before this is now driven from the notifications history table
- Removed from updates and create
- Signatures changes to removed unused params hits many files
- Also potential issue around rate limiting - we used to get the number sent per day from the stats table - which was a single row lookup, now we have to count this. This applies to EVERY API CALL. Probably not a good thing and should be addressed urgently.
Although using a team key is functionally the same as your service being
restricted, conflating the two errors is not helpful. What we typically
saw in research was that someone was using a team key, got the error,
used a live key and got the _same_ error.
This commit adds a new error message that specifically mentions the type
of API key that you’re using.
Scenario we saw in research:
- trying to send a message to someone outside your team
- service is in trial mode
Result:
- error message was terrible, no-one understood it
Solution:
- better error message
Removed all existing statsd logging and replaced with:
- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables
- statsd client changed to prefix all stats with "notification.api."
- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way
notifications.api.POST.notifications.send_notification.200
This should allow us to aggregate to the level of
- API or ADMIN
- POST or GET etc
- modules
- methods
- status codes
Finally we count the callbacks received from 3rd parties to mapped status.
"with personalisation" should only be used by the public notification api
"with template" should be used when we want template name, etc details.
also added an xfail test for correctly constructing notification
personalisation
rename the notification_status_schema to make it apparent that it
involves the template, and then don't use it on the job page - the
job page doesn't do anything with the data. won't somebody think of
the cpu cycles! (also means it ignores problems with template
versions)
if api_key used to access endpoint is type team, endpoints only return that type -
will 404 if you provide a different ID. Same applies for normal (normal api keys
cannot see team notifications)
also, for convenience, set sample_notification to supply key_type of KEY_TYPE_NORMAL
by default
moved from notifications/rest -> service/rest and job/rest respectively
endpoint routes not affected
removed requires_admin decorator - that should be set by nginx config
as opposed to python code
notifications, when retrieved by notification id, or service id (i.e.
all notifications for service).
There is a new element returned at top level of notification json called
body, which is the template content merged with personalisation. This
is consistent with api to endpoint to create notification which returns
what was sent as 'body' in json response.
Merging of template with personalisation is done in the
NotificationStatusSchema.
Personalisation data in encrypted before storing in db.
or param errors to raise invalid data exception. That will cause
those responses to be handled in by errors.py, which will log
the errors.
Set most of schemas to strict mode so that marshmallow will raise
exception rather than checking for errors in return tuple from load.
Added handler to errors.py for marshmallow validation errors.
- if a service is in research mode the don't send the notifications via the providers (MMG/SES/etc)
- instead set up a task to mimic those services callbacks - this completes the loop, and show stats, delivery receipts and so on.
- Use the "to" field to choose the response, allows users to create successful and errored notifications
temp fail sms, uses "07833333333"
perm fail sms, uses = "07822222222"
success = "07811111111" (or anything else)
success email = "delivered@simulator.notify"
perm fail = "perm-fail@simulator.notify"
temp fail = "temp-fail@simulator.notify"
