Commit Graph

85 Commits

Author SHA1 Message Date
Martyn Inglis
893164ae40 Removed contented updates the notifications stats table
- As before this is now driven from the notifications history table

- Removed from updates and create
- Signatures changes to removed unused params hits many files
- Also potential issue around rate limiting - we used to get the number sent per day from the stats table - which was a single row lookup, now we have to count this. This applies to EVERY API CALL. Probably not a good thing and should be addressed urgently.
2016-08-25 11:55:38 +01:00
Martyn Inglis
708f566c24 Removed updates to the provider stats table
- again these new come from the notifications history table
- We update this when we sent a notification, so removed from celery tasks
- tests removed also
2016-08-25 10:33:12 +01:00
Martyn Inglis
84ea173ced Removed updates to templates statistics
- on create notification we updated the templates stats to record the usage.
- this is now based on notification history
- this update and associated tests are now removed,
2016-08-25 10:12:39 +01:00
Martyn Inglis
44bc071037 Removed the updates to the job table to track delivery of notifications
Previously we kept a running total of job progress/success/failure on the job table. This causes contention, we now generate this data from notification history.

Removed these updates.
2016-08-25 09:29:53 +01:00
Leo Hemsted
1e8ab35976 Merge branch 'master' into noti-stats-cleanup 2016-08-24 14:24:17 +01:00
Leo Hemsted
132addb181 Merge branch 'master' into noti-stats-cleanup 2016-08-23 09:49:12 +01:00
Martyn Inglis
4e6da1ba55 Changed template stats for template id
- now returns the most recent notification history row for that template ID.
- contains all the required data for the use cases for that template
2016-08-22 14:35:04 +01:00
Martyn Inglis
fdd85b7dc0 Merge branch 'master' into template_queryies 2016-08-22 11:42:26 +01:00
Martyn Inglis
b7476a1975 Removed the group by day aspects of template stats. Not needed. Grouped by template only. 2016-08-22 10:38:44 +01:00
Martyn Inglis
f74000f548 Add query to get template usage from the Notifications History table
- groups by template Id and Day.

Returns count per day, template name, template id, template type, and day.

Ordered by day (desc) and template name (acc)
2016-08-18 12:06:00 +01:00
Leo Hemsted
f065b08db2 remove unused dao functions 2016-08-11 16:13:53 +01:00
Leo Hemsted
46c0728b12 add actual_template relationship to notification
also renamed the function to make it apparent that it'll join and grab personalisation
2016-08-09 16:57:18 +01:00
minglis
d67e43a6d0 Merge pull request #583 from alphagov/stats-db-updates
Stats db updates
2016-08-08 11:48:01 +01:00
Leo Hemsted
527a5c4eaa calculate billable units when sending an sms
don't calculate it if we're in research mode
* added tests to prove this
* removed last code referring to content_char_count
2016-08-03 16:46:05 +01:00
Martyn Inglis
61aaa36f9b Rewiring how we do statsd
- decorater added to the DAO for notifications
2016-08-02 14:23:47 +01:00
Leo Hemsted
4ef084464d update notification_history table from notification_dao create/update functions
please ensure that any changes to notifications table happen through either dao_create_notification or dao_update_notification.
changed the notification status update triggered by the provider callbacks to ensure that sets updated_by and can update the history table.
also re-added the character_count so we can reconstruct billing data if needed.
2016-07-12 13:28:45 +01:00
Leo Hemsted
722699a72a Update notification_history table on insert/update of notifications
triggered via calls in dao_create_notification and dao_update_notification - if you don't use those functions (eg update in bulk) you'll have to update the history table yourself!
2016-07-12 13:27:31 +01:00
Leo Hemsted
f992240192 make notification.key_type not nullable
set to 'normal' for all existing notifications, and all job notifications also created as 'normal' - so if your eg reporting service hits notify, it gets notifications created from both API calls and front-end csv jobs.
2016-07-01 16:36:11 +01:00
Rebecca Law
60e159e3c0 Add notification_type to notification table.
It seems like an oversight not to include the notification type in the notifcation.
When updating statistics a query to the template table is required to get the type, this update will mean that query does not have to happen.
2016-06-29 11:23:02 +01:00
Rebecca Law
8a0211b3eb Only send to the provider if the notification has a created status.
If the notification ends up in the retry queue and the delivery app is restarted the notification will get sent twice.
This is because when the app is restarted another message will be in the retry queue as message available which is a
duplicate of the one in the queue that is a  message in flight.

This should complete https://www.pivotaltracker.com/story/show/121844427
2016-06-27 14:47:20 +01:00
Rebecca Law
02e37dee3a Merge branch 'master' into notification-created-status 2016-06-22 16:58:33 +01:00
Leo Hemsted
4ce26dc1a4 Merge pull request #463 from alphagov/test-refactor
Remove db.drop_all from notify_db session fixture to speed up tests
2016-06-22 16:53:06 +01:00
Rebecca Law
11ac7b2874 Merge branch 'master' into notification-created-status
Conflicts:
	app/dao/notifications_dao.py
2016-06-22 16:15:56 +01:00
Leo Hemsted
93618d2a7e make test db return order-agnostic 2016-06-22 15:56:41 +01:00
Rebecca Law
23dfa2d535 Added created as a status type that can be updated. 2016-06-22 14:42:55 +01:00
Rebecca Law
3d3bff25a8 [WIP] updating notification to start in the created status 2016-06-21 15:51:30 +01:00
Martyn Inglis
838bf1d7d4 Merge branch 'master' into split-sms-and-retry
Conflicts:
	app/celery/tasks.py
	tests/app/celery/test_tasks.py
2016-06-17 16:39:03 +01:00
Chris Hill-Scott
7f7743b389 Test for filtering a job by notification status
It’s going to be useful to see all the notifications for a job that are
failed/delivered/etc.

The API seems to support this behaviour already, but it doesn’t seem to
be tested.

This commit adds some testsfor the DAO and REST layers.
2016-06-14 14:07:54 +01:00
Martyn Inglis
6b5b40b953 Added tests for statsd and provider stats outcomes with the new provider stats tasks
- note large change to DAO to remove provider from create notification. Added on send now not creation.
2016-06-13 11:38:25 +01:00
Martyn Inglis
5e1613e461 pep8 too many lines 2016-06-02 10:35:54 +01:00
Martyn Inglis
c6c534365e Fixing tests as stats update now don't update reference 2016-06-02 10:00:27 +01:00
Martyn Inglis
754ccbe9af Removed update reference from updating the provider stats
- single focus method
- allows not to pollute DAO with research mode
2016-06-02 09:52:47 +01:00
Nicholas Staples
276ca15919 Provider stats only updated if the provider successfully sends the message. 2016-06-01 12:43:26 +01:00
Martyn Inglis
2108bb52d5 Reorder updates to the notification / stats tables to match create. 2016-05-31 10:17:15 +01:00
Rebecca Law
25a1b7f31c Firetext does not have a status code for temporary-failure.
In order to set a message as temporary-failure, we check if it is in pending status first.
Otherwise a delivery receipt for failure is set to permanent failure.
2016-05-26 16:46:00 +01:00
Rebecca Law
d6c7f03607 Added another assert to the test 2016-05-24 16:37:05 +01:00
Rebecca Law
e4eecb894a Update job.notifications_delivered and job.notifications_failed count when updating the status of a notification. 2016-05-24 11:31:44 +01:00
Rebecca Law
cf2723bdc9 If the sms client provider sends more than one delivery receipt only take the update for the the first one.
Only update the notification.status if status = sending.
2016-05-20 17:04:56 +01:00
Rebecca Law
281a93b32d Update ses callback to interpret hard and soft bounces.
If the notification has a status == sending then update the status otherwise do not update the status.
In other words do not change the status more than once.
2016-05-17 15:38:49 +01:00
Rebecca Law
917110870d Use the template version at the time the notification is created or at the time the job is created.
Update notifications/sms|email endpoint to send the template version to the queue.
Update the process_job celery talk to send the template version to the queue.
When the send_sms|send_email task runs it will get the template by id and version.

Created a data migration script to add the template_vesion column for jobs and notifications.
The existing jobs and notifications are given the template_version of the current template.
There is a chance this is the wrong template version, but deemed okay since the application is not live.

Create unit test for the dao_get_template_versions method.
Rename /template/<id>/version to /template/<id>/versions which returns all versions for that template id and service id.
2016-05-13 16:25:05 +01:00
Rebecca Law
c8c0f95dd2 Merge branch 'master' into add-template-version
Conflicts:
	tests/app/dao/test_notification_dao.py
2016-05-12 09:49:35 +01:00
Rebecca Law
f223da04df [WIP] Fix for dao test 2016-05-12 09:20:49 +01:00
Rebecca Law
f72f5aba05 [WIP]
Start to add template_version to jobs and notification
2016-05-11 17:04:51 +01:00
Martyn Inglis
57e05feafb Updated the Provider stats and rates DAO objects to query based on the identifier in the ProviderDetails object.
- updated all tests
- changed teardown to leave provider details rows on end of individual tests
2016-05-06 09:09:47 +01:00
Nicholas Staples
f71dbe9c0f Message limit added and all tests passing. 2016-04-29 10:36:59 +01:00
Adam Shimali
4e81695c84 Added limit_days filter to get notifications dao. 2016-04-28 16:51:19 +01:00
Chris Hill-Scott
d7f6bb9976 Align date range for template + notification stats
We were using two different queries to filter template stats to the past
7 days, plus today.

Since we’re storing both as short dates, we can now use the same query
for both.

0      | 1       | 2         | 3        | 4      | 5        | 6      | 7
-------|---------|-----------|----------|--------|----------|--------|-------
Monday | Tuesday | Wednesday | Thursday | Friday | Saturday | Sunday | Monday

So if we are on Monday, the stats should include today, plus everything
back to last Monday.

Previously the template stats query was only going back to the Tuesday.

This should mean the numbers on the dashboard always line up.
2016-04-27 11:11:09 +01:00
Adam Shimali
886f302993 Removed unused code 2016-04-25 17:21:55 +01:00
Adam Shimali
24ea6f1637 Use short dates when selection notifications for deletion.
This means we will retain notifications for a full week and not
delete records that are 7 x 24 hours older than the time of the run of
the deletion task.

Also the task only needs to run once a day now, so I have changed
the celery config for the deletion tasks.
2016-04-25 16:12:46 +01:00
Nicholas Staples
3b1423a2ea Provider Statistics added.
Rates command added with a test.

Updated to include added migration.
2016-04-21 13:47:04 +01:00