The financial year start April 1, 00:00 BST and our dates are stored as UTC.
Added a test for get_april_fools.
Added some test more test data for get_billable_unit_count_per_month.
* to be used for auditing changes to provider details
* not hooked in yet
* also made provider_details.active non-nullable. set all existing null
provider_details.active to FALSE. none are set false on live so
should hopefully be fairly uncontroversial
* refactored NotificationHistory.from_notification and update from noti
to share with provider_details_history
- note this is an unexpectedly big change.
- When we create a service we pass the service id to the persist method. This means that we don't have the service available to check if in research mode.
- All calling methods (expecting the one where we use the notify service) have the service available. So rather than reload it I changed the method signature to pass the service, not the ID to persist.
- Touches a few places.
Note this means that the update or create methods will fall over on a null service. But this seems correct.
Goes back to the story which we need to play to make the service available as the API user so that the need to load and pass around services is minimised.
Return multiple notifications for a service.
Choosing a page_size or a page_number is no longer allowed.
Instead, there is a `next` link included with will return the
next {default_page_size} notifications in the sequence.
Query parameters accepted are:
- template_type: filter by specific template types
- status: filter by specific statuses
- older_than: return a chronological list of notifications older
than this one. The notification with the id that is passed in
is _not_ returned.
Note that both `template_type` and `status` can accept multiple
parameters. Thus it is possible to call
`/v2/notifications?status=created&status=sending&status=delivered`
- It would be nice to refactor the send_sms and send_email tasks to use these common functions as well, that way I can get rid of the new Notifications.from_v2_api_request method.
- Still not happy with the format of the errors. Would like to find a happy place, where the message is descript enough that we do not need external documentation to explain the error. Perhaps we still only need documentation to explain the trial mode concept.
Notifications with a `billable_units` count of `0` wont have any effect
on the result, but including them in the query will slow down the
grouping and summing of the results because it’ll have to loop over more
rows.
April 1st is in British summer time, ie 1hr ahead of UTC. The database
stores everything in UTC, so for accurate comparisions we need to make
sure that `get_financial_year()` returns a UTC, datetime-aware
timestamp that is 1hr ahead of midnight.
This also means that when we group notifications by month, the months
need to be in BST. So the line between one year and another is actually
01:00 on April 1st, _not_ 00:00 on April 1st.
There’s no way we’ve found to do this in SQLAlchemy or raw Postgres,
especially because we don’t store the timestamps with a timezone in the
database.
So the grouping and summing of the notifications has to be done in
Python.
In order to invoice people we need to know how many text message
fragments they’ve sent per month.
This should be per (government) financial year, ie April 1st to April
1st because we’ll only ever show a page for one year (because the
250,000 allowance is topped up at the start of every financial year).
This commit only does the DAO bit, not the REST bit.
Developers need visibility of what their integration is doing within
the app. This includes notifications sent with a test key.
This commit adds an optional, defaults-to-false parameter to include
notifications sent from a test API key when getting notifications.
Starting arguments on their own line and putting the closing parenthesis
on it’s own line because any subsequent changes to the arguments diff
cleanly (ie without touching any other lines).
- get all notifications by service
- template usage
- most recently used templates
Ensures that the dashboard shows no test key data. Supplements: https://github.com/alphagov/notifications-api/pull/677 which excludes CSV data. This branches from that so is dependant.
We found that if the notifications were in created or pending they are not purged from notifications.
- New bulk update method to set all notificaitons with:
- a status = created|sending|pending to temporary-failure
- and is older then today minus SENDING_NOTIFICATIONS_TIMEOUT_PERIOD (in seconds)
- the scheduled task to timeout notifications use the new bulk update query.
- the task will be more efficient
- As before this is now driven from the notifications history table
- Removed from updates and create
- Signatures changes to removed unused params hits many files
- Also potential issue around rate limiting - we used to get the number sent per day from the stats table - which was a single row lookup, now we have to count this. This applies to EVERY API CALL. Probably not a good thing and should be addressed urgently.
- again these new come from the notifications history table
- We update this when we sent a notification, so removed from celery tasks
- tests removed also
- on create notification we updated the templates stats to record the usage.
- this is now based on notification history
- this update and associated tests are now removed,
Previously we kept a running total of job progress/success/failure on the job table. This causes contention, we now generate this data from notification history.
Removed these updates.
- groups by template Id and Day.
Returns count per day, template name, template id, template type, and day.
Ordered by day (desc) and template name (acc)
only in the public notification endpoint so far for fear of breaking
things - in an ideal world i'd remove the template relationship
from models entirely and replace that with actual_template
Removed all existing statsd logging and replaced with:
- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables
- statsd client changed to prefix all stats with "notification.api."
- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way
notifications.api.POST.notifications.send_notification.200
This should allow us to aggregate to the level of
- API or ADMIN
- POST or GET etc
- modules
- methods
- status codes
Finally we count the callbacks received from 3rd parties to mapped status.
please ensure that any changes to notifications table happen through either dao_create_notification or dao_update_notification.
changed the notification status update triggered by the provider callbacks to ensure that sets updated_by and can update the history table.
also re-added the character_count so we can reconstruct billing data if needed.
