Commit Graph

194 Commits

Author SHA1 Message Date
Leo Hemsted
132addb181 Merge branch 'master' into noti-stats-cleanup 2016-08-23 09:49:12 +01:00
Martyn Inglis
4e6da1ba55 Changed template stats for template id
- now returns the most recent notification history row for that template ID.
- contains all the required data for the use cases for that template
2016-08-22 14:35:04 +01:00
Martyn Inglis
fdd85b7dc0 Merge branch 'master' into template_queryies 2016-08-22 11:42:26 +01:00
Martyn Inglis
b7476a1975 Removed the group by day aspects of template stats. Not needed. Grouped by template only. 2016-08-22 10:38:44 +01:00
Martyn Inglis
f74000f548 Add query to get template usage from the Notifications History table
- groups by template Id and Day.

Returns count per day, template name, template id, template type, and day.

Ordered by day (desc) and template name (acc)
2016-08-18 12:06:00 +01:00
Leo Hemsted
5b3a0f03d3 rename actual_template to template_history
it's slightly less emotionally charged
2016-08-12 10:29:23 +01:00
Leo Hemsted
f065b08db2 remove unused dao functions 2016-08-11 16:13:53 +01:00
Leo Hemsted
4ba3745159 update schema to use template_history for accurate template details
only in the public notification endpoint so far for fear of breaking
things - in an ideal world i'd remove the template relationship
from models entirely and replace that with actual_template
2016-08-09 16:57:18 +01:00
Leo Hemsted
46c0728b12 add actual_template relationship to notification
also renamed the function to make it apparent that it'll join and grab personalisation
2016-08-09 16:57:18 +01:00
minglis
d67e43a6d0 Merge pull request #583 from alphagov/stats-db-updates
Stats db updates
2016-08-08 11:48:01 +01:00
Martyn Inglis
f223446f73 Refactor statsd logging
Removed all existing statsd logging and replaced with:

- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables

- statsd client changed to prefix all stats with "notification.api."

- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way

	notifications.api.POST.notifications.send_notification.200

This should allow us to aggregate to the level of

	- API or ADMIN
	- POST or GET etc
	- modules
	- methods
	- status codes

Finally we count the callbacks received from 3rd parties to mapped status.
2016-08-05 10:44:43 +01:00
Leo Hemsted
143cfb526c change update_provider_stats to use billable_units
updated tests etc, and removed some old tests that are no longer relevant
2016-08-03 17:22:20 +01:00
Martyn Inglis
61aaa36f9b Rewiring how we do statsd
- decorater added to the DAO for notifications
2016-08-02 14:23:47 +01:00
Leo Hemsted
4ef084464d update notification_history table from notification_dao create/update functions
please ensure that any changes to notifications table happen through either dao_create_notification or dao_update_notification.
changed the notification status update triggered by the provider callbacks to ensure that sets updated_by and can update the history table.
also re-added the character_count so we can reconstruct billing data if needed.
2016-07-12 13:28:45 +01:00
Leo Hemsted
722699a72a Update notification_history table on insert/update of notifications
triggered via calls in dao_create_notification and dao_update_notification - if you don't use those functions (eg update in bulk) you'll have to update the history table yourself!
2016-07-12 13:27:31 +01:00
Rebecca Law
1bfd25107b Add a lockmode for the update_notifications_status_by_id to prevent the timeout task from updating the same notification more than once.
This happens because more than one beat process was creating the timeout task, resulting in multiple workers running the same queries at the same time.
2016-07-08 14:48:07 +01:00
Leo Hemsted
2cf1d22748 add filters to GET /notifications endpoints to only return for provided key_type
if api_key used to access endpoint is type team, endpoints only return that type -
will 404 if you provide a different ID. Same applies for normal (normal api keys
cannot see team notifications)
also, for convenience, set sample_notification to supply key_type of KEY_TYPE_NORMAL
by default
2016-07-01 14:35:02 +01:00
Rebecca Law
1ce3495fc7 Remove reference to template.template_type for the stats update query.
Removes a query during the stats update.
2016-06-29 12:12:35 +01:00
Rebecca Law
25db1bce74 Use the notification types enum for the notifications.notification_type.
Reuse EMAIL_TYPE in template_types and notification_types.
2016-06-29 11:50:54 +01:00
Rebecca Law
23dfa2d535 Added created as a status type that can be updated. 2016-06-22 14:42:55 +01:00
Martyn Inglis
bec882ba41 Merge branch 'master' into split-sms-and-retry 2016-06-13 11:39:13 +01:00
Martyn Inglis
6b5b40b953 Added tests for statsd and provider stats outcomes with the new provider stats tasks
- note large change to DAO to remove provider from create notification. Added on send now not creation.
2016-06-13 11:38:25 +01:00
Leo Hemsted
3e72440f38 fix template api tests being inconsistent by adding ordering 2016-06-09 11:24:04 +01:00
Nicholas Staples
8a1f4de217 Task added to update 'sending' notifications after 72 hours, set task to temporary-failure. 2016-06-08 15:25:57 +01:00
Leo Hemsted
e3d9dfad6e add template_statistics endpoint for specific template
`/service/<service_id>/template-statistics/<template_id>`
still requires service-id just to try and keep api tree cleaner
2016-06-07 14:18:42 +01:00
Martyn Inglis
12503d6291 First spike to split up send-sms task
- 2 separate tasks - DB and Provider
- DB to persist notification
- Provider to contact provider
- Each piece has separate retries
- Provider retries have configured back-off
2016-06-03 14:54:46 +01:00
Martyn Inglis
754ccbe9af Removed update reference from updating the provider stats
- single focus method
- allows not to pollute DAO with research mode
2016-06-02 09:52:47 +01:00
Martyn Inglis
49f386e7c5 Refactored to put notifications update last 2016-06-02 09:17:06 +01:00
Nicholas Staples
276ca15919 Provider stats only updated if the provider successfully sends the message. 2016-06-01 12:43:26 +01:00
Martyn Inglis
2108bb52d5 Reorder updates to the notification / stats tables to match create. 2016-05-31 10:17:15 +01:00
Rebecca Law
2bfa9b7e10 Merge branch 'master' into permanent-failure-firetext 2016-05-27 12:10:59 +01:00
Rebecca Law
885abcaf16 Make update notification status transcational.
Refactor methods for reusability.
2016-05-27 12:09:36 +01:00
Leo Hemsted
a90a18541f notifications/statistics returns when no stats exist for today
moved filtering from WHERE to JOIN ON so that when join suceeds but filter fails, we dont lose the service's data from the results set
2016-05-26 16:48:30 +01:00
Rebecca Law
25a1b7f31c Firetext does not have a status code for temporary-failure.
In order to set a message as temporary-failure, we check if it is in pending status first.
Otherwise a delivery receipt for failure is set to permanent failure.
2016-05-26 16:46:00 +01:00
Rebecca Law
83df16025d Merge pull request #347 from alphagov/get-jobs-created-in-last-7-days
Add a limit days query param for get all jobs.
2016-05-25 11:36:31 +01:00
Rebecca Law
a67793cf8a Updated date query so that the created_at date is truncated so that we get all jobs created number of days requested. 2016-05-25 11:13:49 +01:00
Rebecca Law
e4eecb894a Update job.notifications_delivered and job.notifications_failed count when updating the status of a notification. 2016-05-24 11:31:44 +01:00
Rebecca Law
cf2723bdc9 If the sms client provider sends more than one delivery receipt only take the update for the the first one.
Only update the notification.status if status = sending.
2016-05-20 17:04:56 +01:00
Leo Hemsted
81e5eb2c0a return all zeros when NotificationStatistics dont exist 2016-05-20 15:24:57 +01:00
Leo Hemsted
0ad050043c new /notification/statistics endpoint 2016-05-20 09:48:06 +01:00
Nicholas Staples
0fe0c1d2b4 Added job row number to the notification for csv jobs. All tests passing. 2016-05-19 10:46:03 +01:00
Rebecca Law
281a93b32d Update ses callback to interpret hard and soft bounces.
If the notification has a status == sending then update the status otherwise do not update the status.
In other words do not change the status more than once.
2016-05-17 15:38:49 +01:00
Martyn Inglis
8669a169eb Merge branch 'master' into primary-provider
Conflicts:
	app/dao/notifications_dao.py
	app/dao/provider_statistics_dao.py
	app/schemas.py
	tests/app/conftest.py
2016-05-10 09:13:02 +01:00
NIcholas Staples
f06aaa924e Merge pull request #287 from alphagov/statistics_group_by_week_backup
Group by Notification Statistics added and all tests working.
2016-05-09 10:17:37 +01:00
Nicholas Staples
d40d9f76d4 Purge functional tests command added with passing test.
Code quality fix.
2016-05-06 11:16:30 +01:00
Martyn Inglis
57e05feafb Updated the Provider stats and rates DAO objects to query based on the identifier in the ProviderDetails object.
- updated all tests
- changed teardown to leave provider details rows on end of individual tests
2016-05-06 09:09:47 +01:00
Nicholas Staples
f465066a69 Group by Notification Statistics added and all tests working.
Fixed tests for upgraded client.
2016-05-05 11:04:20 +01:00
Nicholas Staples
f71dbe9c0f Message limit added and all tests passing. 2016-04-29 10:36:59 +01:00
Adam Shimali
4e81695c84 Added limit_days filter to get notifications dao. 2016-04-28 16:51:19 +01:00
Chris Hill-Scott
d7f6bb9976 Align date range for template + notification stats
We were using two different queries to filter template stats to the past
7 days, plus today.

Since we’re storing both as short dates, we can now use the same query
for both.

0      | 1       | 2         | 3        | 4      | 5        | 6      | 7
-------|---------|-----------|----------|--------|----------|--------|-------
Monday | Tuesday | Wednesday | Thursday | Friday | Saturday | Sunday | Monday

So if we are on Monday, the stats should include today, plus everything
back to last Monday.

Previously the template stats query was only going back to the Tuesday.

This should mean the numbers on the dashboard always line up.
2016-04-27 11:11:09 +01:00