Commit Graph

1149 Commits

Author SHA1 Message Date
Martyn Inglis
e6347d99bd Merge branch 'master' into stats-db-updates
Conflicts:
	tests/app/conftest.py
2016-08-03 11:46:40 +01:00
Martyn Inglis
61aaa36f9b Rewiring how we do statsd
- decorater added to the DAO for notifications
2016-08-02 14:23:47 +01:00
Leo Hemsted
444132ad66 rewrite weekly aggregate function to honor week boundaries
(and not use the stats table, and also be easier to read)
2016-07-26 17:23:59 +01:00
Leo Hemsted
fd96c854e1 add dao function to get notification stats for a server for toady only 2016-07-26 14:35:29 +01:00
Leo Hemsted
2d1babf2bb 'detailed' flag on GET /service/<uuid>
if passed in, returns the service object with additional statistics
dictionary, which will be used in the admin app to populate dashboard
components. A new schema has been created for this to avoid clashing/
causing confusion with the existing schema, which is already used
for PUT/POST as well, and this schema can be easily tailored to
reduce ambiguity and lazy-loading
2016-07-21 14:04:25 +01:00
Leo Hemsted
566ae798df Speed up GET services endpoints
Add additional relationships to exclude in the ServiceSchema metaclass.
Marshmallow by default lazily loads relationships when dumping, so any
relationships we know we won't need, we can exclude and avoid a DB call.
Lots of tables are linked to services, so it loads a lot of tables.
So don't load statistics tables, since they're clearly not needed.

We *do* however want to return the users for the service - they're used
in a few places. If we're returning all services, then we don't want to
make separate queries for these users, so we modify the services_dao
queries to load users the first time round. This should speed up all
GET queries to the services endpoints, most notably pages that get many
services (platform_admin, choose service, login)
2016-07-15 12:04:38 +01:00
Leo Hemsted
4ef084464d update notification_history table from notification_dao create/update functions
please ensure that any changes to notifications table happen through either dao_create_notification or dao_update_notification.
changed the notification status update triggered by the provider callbacks to ensure that sets updated_by and can update the history table.
also re-added the character_count so we can reconstruct billing data if needed.
2016-07-12 13:28:45 +01:00
Leo Hemsted
722699a72a Update notification_history table on insert/update of notifications
triggered via calls in dao_create_notification and dao_update_notification - if you don't use those functions (eg update in bulk) you'll have to update the history table yourself!
2016-07-12 13:27:31 +01:00
Rebecca Law
1bfd25107b Add a lockmode for the update_notifications_status_by_id to prevent the timeout task from updating the same notification more than once.
This happens because more than one beat process was creating the timeout task, resulting in multiple workers running the same queries at the same time.
2016-07-08 14:48:07 +01:00
Leo Hemsted
2cf1d22748 add filters to GET /notifications endpoints to only return for provided key_type
if api_key used to access endpoint is type team, endpoints only return that type -
will 404 if you provide a different ID. Same applies for normal (normal api keys
cannot see team notifications)
also, for convenience, set sample_notification to supply key_type of KEY_TYPE_NORMAL
by default
2016-07-01 14:35:02 +01:00
Leo Hemsted
adbe02783d refactor authentication code
moved api_key secret manipulation (generating and getting) into
authentiation/utils, and added a property on the model, to facilitate
easier matching of authenticated requests and the api keys they used
2016-06-30 10:44:21 +01:00
Rebecca Law
1ce3495fc7 Remove reference to template.template_type for the stats update query.
Removes a query during the stats update.
2016-06-29 12:12:35 +01:00
Rebecca Law
25db1bce74 Use the notification types enum for the notifications.notification_type.
Reuse EMAIL_TYPE in template_types and notification_types.
2016-06-29 11:50:54 +01:00
Rebecca Law
5f29815870 Merge pull request #464 from alphagov/refactor-api-key
Refactor the api_key_dao.
2016-06-22 15:57:38 +01:00
Rebecca Law
acee87fc63 Refactor the api_key_dao.
The only update we should be doing to an api key is to expire/revoke the api key.
Removed the update_dict from the the save method.
Added an expire_api_key method that only updates the api key with an expiry date.
2016-06-22 15:27:28 +01:00
Rebecca Law
23dfa2d535 Added created as a status type that can be updated. 2016-06-22 14:42:55 +01:00
Martyn Inglis
838bf1d7d4 Merge branch 'master' into split-sms-and-retry
Conflicts:
	app/celery/tasks.py
	tests/app/celery/test_tasks.py
2016-06-17 16:39:03 +01:00
Rebecca Law
f8262b13c1 Refactor permissions dao - removed custom query filter
Removed permissions rest - not being used.
2016-06-15 16:32:52 +01:00
Martyn Inglis
bec882ba41 Merge branch 'master' into split-sms-and-retry 2016-06-13 11:39:13 +01:00
Martyn Inglis
6b5b40b953 Added tests for statsd and provider stats outcomes with the new provider stats tasks
- note large change to DAO to remove provider from create notification. Added on send now not creation.
2016-06-13 11:38:25 +01:00
Leo Hemsted
3e72440f38 fix template api tests being inconsistent by adding ordering 2016-06-09 11:24:04 +01:00
Nicholas Staples
8a1f4de217 Task added to update 'sending' notifications after 72 hours, set task to temporary-failure. 2016-06-08 15:25:57 +01:00
Adam Shimali
e28ef237e4 When adding a user new with permissions to a service, the permissions
dao was deleting all permissions for that user (regardless of service
id) as the last filter on the permissions dao get_query method won.

I've added a replace flag to the set_user_service_permission method
so that it can handle adding new users + permissions and editing
of existing users' permissions.

Also by pass the get_query method until it can be refactored to work
correctly.

For now execute the filter query directly on the model.
2016-06-07 16:35:26 +01:00
Leo Hemsted
e3d9dfad6e add template_statistics endpoint for specific template
`/service/<service_id>/template-statistics/<template_id>`
still requires service-id just to try and keep api tree cleaner
2016-06-07 14:18:42 +01:00
Martyn Inglis
37fe2342ae Merge branch 'master' into split-sms-and-retry 2016-06-07 11:10:17 +01:00
Adam Shimali
63c29a3a3d API recieves full set of permissions names on create of user
invite. This is instead of mapping from permission groups to individual
permissions on user creation.
2016-06-06 12:37:06 +01:00
Chris Hill-Scott
c2ae4773da Order templates by newest created first
When you add a new template, it’s probably the one that you want to do
subsequent stuff with. But it’s also helpful to see the template in
context (with its siblings) to understand that there are multiple
templates. So we don’t want to do what we do in
https://github.com/alphagov/notifications-admin/pull/648
for adding a new template.

But we _can_ make your brand-new template appear first by always
ordering by when the template was created.

This also removes the confusion caused by having `updated_at` affecting
order, and causing the templates to move around all the time.
2016-06-06 10:44:40 +01:00
Martyn Inglis
12503d6291 First spike to split up send-sms task
- 2 separate tasks - DB and Provider
- DB to persist notification
- Provider to contact provider
- Each piece has separate retries
- Provider retries have configured back-off
2016-06-03 14:54:46 +01:00
Martyn Inglis
754ccbe9af Removed update reference from updating the provider stats
- single focus method
- allows not to pollute DAO with research mode
2016-06-02 09:52:47 +01:00
Martyn Inglis
001149c263 Merge branch 'master' into research-mode-implementation 2016-06-02 09:28:35 +01:00
Martyn Inglis
49f386e7c5 Refactored to put notifications update last 2016-06-02 09:17:06 +01:00
Nicholas Staples
276ca15919 Provider stats only updated if the provider successfully sends the message. 2016-06-01 12:43:26 +01:00
Martyn Inglis
9617f0748b Added some tests around creating and updated services
- ensure research mode is respected on creation and update
- ensure rest client gives an error for bad research mode update
2016-05-31 12:49:06 +01:00
Martyn Inglis
2108bb52d5 Reorder updates to the notification / stats tables to match create. 2016-05-31 10:17:15 +01:00
Rebecca Law
2bfa9b7e10 Merge branch 'master' into permanent-failure-firetext 2016-05-27 12:10:59 +01:00
Rebecca Law
885abcaf16 Make update notification status transcational.
Refactor methods for reusability.
2016-05-27 12:09:36 +01:00
Leo Hemsted
a90a18541f notifications/statistics returns when no stats exist for today
moved filtering from WHERE to JOIN ON so that when join suceeds but filter fails, we dont lose the service's data from the results set
2016-05-26 16:48:30 +01:00
Rebecca Law
25a1b7f31c Firetext does not have a status code for temporary-failure.
In order to set a message as temporary-failure, we check if it is in pending status first.
Otherwise a delivery receipt for failure is set to permanent failure.
2016-05-26 16:46:00 +01:00
Rebecca Law
83df16025d Merge pull request #347 from alphagov/get-jobs-created-in-last-7-days
Add a limit days query param for get all jobs.
2016-05-25 11:36:31 +01:00
Rebecca Law
a67793cf8a Updated date query so that the created_at date is truncated so that we get all jobs created number of days requested. 2016-05-25 11:13:49 +01:00
Rebecca Law
2a7872cce1 Add a limit days query param for get all jobs. 2016-05-24 17:21:04 +01:00
Rebecca Law
6735a7103c Merge pull request #345 from alphagov/notification-job-status-counts
Notification job status counts
2016-05-24 16:47:00 +01:00
Rebecca Law
e4eecb894a Update job.notifications_delivered and job.notifications_failed count when updating the status of a notification. 2016-05-24 11:31:44 +01:00
Leo Hemsted
5ebae9c66c only return unarchived templates when getting all templates
get_specific_template functions happily return archived templates
2016-05-23 14:20:02 +01:00
Rebecca Law
cf2723bdc9 If the sms client provider sends more than one delivery receipt only take the update for the the first one.
Only update the notification.status if status = sending.
2016-05-20 17:04:56 +01:00
Rebecca Law
a8c79bf80d Merge pull request #334 from alphagov/case-insensitive-email
Make the get_user_by_email_address case insensitive.
2016-05-20 15:55:38 +01:00
Leo Hemsted
81e5eb2c0a return all zeros when NotificationStatistics dont exist 2016-05-20 15:24:57 +01:00
Rebecca Law
0f9a019945 Make the get_user_by_email_address case insenstive. 2016-05-20 13:43:31 +01:00
Leo Hemsted
0ad050043c new /notification/statistics endpoint 2016-05-20 09:48:06 +01:00
Nicholas Staples
0fe0c1d2b4 Added job row number to the notification for csv jobs. All tests passing. 2016-05-19 10:46:03 +01:00