Commit Graph

2708 Commits

Author SHA1 Message Date
Katie Smith
a87be9b74a Use new value of SMS_CHAR_COUNT_LIMIT from utils
Admin, API and utils were all defining a value for SMS_CHAR_COUNT_LIMIT.
This value has been updated in notifications-utils to allow text
messages to be 4 fragments long and notifications-api now gets the value of
SMS_CHAR_COUNT_LIMIT from notifications-utils instead of defining it in
config.

Also updated some tests to check for the higher limit.
2018-08-16 16:34:34 +01:00
Alexey Bezhan
1e3004ab3a Add service stats dao tests with variable days limit 2018-08-15 11:10:16 +01:00
Alexey Bezhan
ab428048b9 Add an endpoint to fetch service data retention by type
Admin app needs to get the service data retention for the specified
notification type, so to avoid iterating through the list of all
existing service data retention settings we restore the endpoint
to get the individual data retention period.
2018-08-13 16:45:57 +01:00
Alexey Bezhan
5f2724c429 Add limit_days argument to notification statistics endpoint
Allows getting notification counts for a given number of days to
support services with custom data retention periods (admin dashboard
page should still display counts for the last 7 days, while the
notifications page displays all stored notifications).
2018-08-13 16:45:57 +01:00
Rebecca Law
3e51b69df6 Merge pull request #2037 from alphagov/fix-email-branding
This PR fixes the check for the text key in the post data.
2018-08-13 15:17:02 +01:00
Rebecca Law
f965322f25 Fixes to the delete letter notifications.
If there are no files to delete we won't get an excpetion.
Wrap the delete file in a try/except to avoid stopping the entire task.
Fix the missing slash for the file name.
2018-08-13 14:09:51 +01:00
Rebecca Law
2a7f3faec9 Merge branch 'master' into fix-delete-s3 2018-08-13 11:44:44 +01:00
Rebecca Law
eb2c878edd Fix bug with deleting the S3 file.
Removed the duplicate method.
2018-08-13 11:33:19 +01:00
Rebecca Law
69ab649256 Use the EmailBranding.text for the bannder instead of the name. 2018-08-13 10:29:06 +01:00
Rebecca Law
febc6f4e23 This PR fixes the check for the text key in the post data. 2018-08-10 17:27:58 +01:00
Rebecca Law
ebb43082d5 Deal with letters that have not been sent, this can be test letters or letters in tech-failure. 2018-08-10 16:22:25 +01:00
Rebecca Law
85a0fd3b46 Merge pull request #2031 from alphagov/fix_data_retention
Use days of retention when deleting notifications in the nightly job.
2018-08-10 15:35:42 +01:00
Leo Hemsted
62d70dbd4d use contextmanager freeze time instead of decorator
it doesn't appear to work nicely with pytest fixtures in the newer
versions of pytest
2018-08-09 16:47:35 +01:00
Rebecca Law
d0e9ab4972 If the notifications that are being deleted are letters then we need to delete the letter from s3 as well. 2018-08-08 16:20:25 +01:00
Rebecca Law
43000b2253 Add a new test 2018-08-06 15:13:08 +01:00
Rebecca Law
517608dff5 Update method to delete notifications that are a week old, to look at the days of retention set for the service.
If the service does not have the days of retention set, then use 7 days.
Added a method to get days of retention for a service and notificaiton type
2018-08-06 13:51:54 +01:00
Rebecca Law
019c6a4e3a Revert "Purge notifications for configured days of retention" 2018-08-03 14:35:36 +01:00
Rebecca Law
78f8c17fe8 Merge pull request #1965 from alphagov/check-data-retention-before-purging-notifications
Purge notifications for configured days of retention
2018-08-03 13:49:49 +01:00
Rebecca Law
5c52679685 Merge pull request #1989 from alphagov/email_branding_name_and_text
Email branding name and text
2018-08-02 16:32:44 +01:00
Rebecca Law
c30e58eaad Instead of deleting the data in the name field just copy it to the text field.
The admin app still needs to use the name column.
Add the text field to the post data schemas.
If the text is not in the post data, then populate it with the data in the name field.
This should make the migration to text easier, and will work until we are able to update the admin app.
2018-08-02 15:55:45 +01:00
Athanasios Voutsadakis
56ba5d207b Refactor test fix 2018-08-01 11:31:53 +01:00
Pea Tyczynska
d8d94b431f Find email_branding by name 2018-07-31 17:58:19 +01:00
Pea Tyczynska
c5792f4d79 Migration file for email_branding fields update 2018-07-31 17:58:19 +01:00
Athanasios Voutsadakis
f14107b723 Fix test that checks queues 2018-07-31 17:26:33 +01:00
Rebecca Law
2f120b30e8 Fix codestyle 2018-07-31 14:13:15 +01:00
Rebecca Law
ab28691a70 Add number to migration file name 2018-07-31 14:06:29 +01:00
Rebecca Law
19890412c9 Added a cancelled notification status so that we can cancel letters before they are sent. 2018-07-31 13:52:04 +01:00
Rebecca Law
67639bf49c Merge branch 'master' into check-data-retention-before-purging-notifications 2018-07-31 11:40:58 +01:00
Rebecca Law
ff2334937c Last night we had some letter notifications that were in permanent failure and had page count = 0.
This causes the create-nighlty-billing task to fail.
This will make sure that 0 is returned in this case.
2018-07-31 11:04:48 +01:00
Pea Tyczynska
ca2b350a99 Remove references to monthly_billing table from api 2018-07-30 11:07:42 +01:00
Pea Tyczynska
c0f309a2a6 Delete scheduled task to populate monthly_billing 2018-07-30 11:06:04 +01:00
Leo Hemsted
a826f6e924 make rebuild-ft-billing-for-day remove unused rows
if the billing data was incorrect and needs to be rebuilt, we should
remove old rows. Previously we were only upserting new rows, but old
no-longer-relevant rows were staying in the database. This commit
makes the flask command remove *all* rows for that service and day,
before inserting the rows from the original notification data after.

This commit doesn't change the existing nightly task, nor does it
change the upsert that happens upon viewing the usage page. In normal
usage, there should never be a case where the number of billable units
for a rate decreases. It should only ever increase, thus, never need to
be deleted
2018-07-26 19:10:30 +01:00
Rebecca Law
a812060915 The unique constraint on SeviceCallbackApi was on service_id only.
Now that we have 2 types of api callbacks the constraint to be on service_id + callback_type.
2018-07-25 14:12:13 +01:00
Rebecca Law
4fc004b00a Increase the number of days we calculate billing from 3 to 10 days.
Log exception if the billing counts for letters are different in the dvla response file than what we collected.
2018-07-24 16:28:30 +01:00
Pea (Malgorzata Tyczynska)
1b21a12b83 Merge pull request #1971 from alphagov/add_task_to_send_complaints_on
Add and call task to send complaints on to service callback APIs
2018-07-24 15:07:09 +01:00
Rebecca Law
0675f09afb Need to be able to query the notification statistics for the right number of days.
If the request is for the big numbers on the activity page, then we need to use the number right number of days.
Added an end point to get the data retention for the service and notification type, which is needed on the activity page to say how long the report is available for.
2018-07-23 09:56:04 +01:00
Rebecca Law
dae29a1b61 Update the query for the notifications activity page to return the data for the days of retention if set. 2018-07-23 09:56:04 +01:00
Rebecca Law
9674005c71 Updated the function to delete notifications over a week old to look at ServiceDataRetention to only delete notifications specified number of days for the notification type. 2018-07-23 09:56:03 +01:00
Leo Hemsted
5c4f3e246c make test dvla response file timestamps in a random order
since there'll be a bunch of threads running functional test tasks at
the same time, there's no point always trying to start from the same
second and then stepping back to the same one-second-back file each
time. Also, this leads us to an increased risk of race conditions.

This change takes the same thirty second range, but shuffles it. The
tests, since they're no longer deterministic, now use a new Matcher
object (w/ credit to alexey) to match any filename from within that
thirty second range
2018-07-20 12:09:00 +01:00
Leo Hemsted
0cfed3f514 create fake letter response files with variables timestamps
when a test letter is created on dev or preview, we upload a file to
the dvla ftp response bucket, to test that our integration with s3
works. s3 triggers an sns notification, which we pick up, and then we
download the file and mark the letters it mentions as delivered.

However, if two tests run at the same time, they'll create the same
file on s3. One will just overwrite the next, and the first letter will
never move into delivered - this was causing functional tests to
intermittently fail.

This commit makes the test letter task check if the file exists - if it
does, it moves back one second and tries again. It tries this thirty
times before giving up.
2018-07-20 12:09:00 +01:00
Pea Tyczynska
3048c05850 Revert notification_id name change in delivery_status service_callback_task data 2018-07-20 11:22:38 +01:00
Katie Smith
af5c4d2872 Include the notification sender name when serializing notifications
The `serialize` method of the Notification model now includes the
`created_by_name`. If a notification was sent as a one-off message this
value will now be the name of the person who sent the notification. If
a notification was sent through the API or by a CSV upload we don't
record the id of the sender, so `created_by_name` will be `None`.

This change affects the data that gets returned from these endpoints:
* /v2/notifications/<notification_id>
* /v2/notifications
2018-07-20 09:37:50 +01:00
Pea Tyczynska
812f4d20dd Send complaints on to service callback APIs using an async task 2018-07-19 16:59:39 +01:00
Katie Smith
b1cfa8942a Add one_off filter when getting all notifications for a service
Added the option to filter by one_off messages to the DAO function
`get_notifications_for_service`. Previously, one-off notifications
were not returned - this has changed so that the default is for
one-off notifications to be returned. Also simplified the `include_jobs`
filter for this function.

The DAO function gets used in 3 places - for the V1 and V2 API endpoints,
which will now start to return one-off messages. It also gets used by
the admin app which needs to pass in `include_one_off=False` to the
`get_all_notifications_for_service` where we don't want one-off
notifications to show, such as the API message log page.
2018-07-18 15:08:06 +01:00
Pea Tyczynska
86978c225a Filter 'get_service_callback_api_for_service' to only get status updates
Also rename it to 'get_service_delivery_status_callback_api_for_service'
2018-07-18 11:36:39 +01:00
Pea Tyczynska
183aa160c6 Add callback_type column to service_callback_api table
Also add service_callback_type table with allowed types
2018-07-17 16:54:00 +01:00
Rebecca Law
9c99e45008 Merge pull request #1945 from alphagov/flexible-data-retention
Flexible data retention
2018-07-17 16:03:18 +01:00
Pea (Malgorzata Tyczynska)
48d1e816ef Merge pull request #1943 from alphagov/admin_platform_users_search
Add data endpoint for finding users by full or partial email
2018-07-16 15:47:23 +01:00
Pea Tyczynska
782a8ab9e7 Add data endpoint for finding users by full or partial email 2018-07-13 15:47:20 +01:00
Rebecca Law
ab695d24d5 Added new endpoints to get service data retention for a service. 2018-07-13 15:18:27 +01:00