Commit Graph

860 Commits

Author SHA1 Message Date
Pea Tyczynska
165b65612e Change how rates are filtered by date to equal or greater 2018-09-25 15:10:16 +01:00
Pea Tyczynska
b3191dbc2a Filter letter rates by post_class in get_rate
Also adjust existing tests.
2018-09-25 15:10:15 +01:00
Rebecca Law
5f73dfdbe9 Added constraints to services.postage: not nullable and can only containt "first" or "second" 2018-09-18 17:41:25 +01:00
Rebecca Law
7200ccb452 Remove print 2018-09-18 16:44:07 +01:00
Rebecca Law
1c0892888f Change the name of letter_class to postage, this will match the domain language used in the documentation.
Updated the tests to answer review comments.
2018-09-18 15:22:08 +01:00
Rebecca Law
79815b4f8a In order to support sending letters by first class we need a way to know how the service wants us to send the letter.
To start with this will be an attribute on the service, at the time the notification is created it will look at Service.letter_class to decide what class to use for the letter.

This PR adds Service.letter class as a nullable column.
Updated the create_service and update_service method to default the value to second.

Subsequent PRs will add the check constraint to ensure we only get first or second in the letter_class column and make that column nullable.
This can't be done all at once because it will cause an error if someone inserts or updates a service during the deploy.
2018-09-17 11:27:43 +01:00
Rebecca Law
f1b04193ca In this PR we remove trigger-letter-pdfs-for-day scheduled task and just call collate_letter_pdfs_for_day instead.
There was a datetime bug in the query which resulted in files not being sent to the postal provider.
The trigger-letter-pdfs-for-day task is no longer needed, so rather than fix the query just call collate_letter_pdfs_for_day directly.
Less code is always better.

Deployment considerations: I realized this is strictly not backwards compatible if the scheduled job is in progress and a task is on the queue that no longer exists. This is ok since we will deploy this well before 17:50.
2018-09-12 17:16:34 +01:00
Alexey Bezhan
18ab7f3337 Add updated history count to dao_update_notifications_by_reference
For returned letter updates most notifications won't exist in the
notifications table, so in order to find out whether the reference
matches any known letters we need to check the count of updated
history records.
2018-09-04 16:55:22 +01:00
Chris Hill-Scott
5d5047fb13 Add unique constraint to email branding domains
Does two things:

1. Revert "Revert "Add unique constraint to email branding domain""

This reverts commit af9cb30ef3.

2. Don’t allow empty string in email branding domain

Columns with multiple `null`s can have a uniqueness constraint. Columns
with multiple empty string values are not considered unique.

This commit:
- removes any duplicate empty string values
- casts empty strings to null string any time these columns are updated

---

Squashed into this single commits because these two things are not
atomic as individual commits.
2018-09-04 11:10:46 +01:00
Alexey Bezhan
5f2724c429 Add limit_days argument to notification statistics endpoint
Allows getting notification counts for a given number of days to
support services with custom data retention periods (admin dashboard
page should still display counts for the last 7 days, while the
notifications page displays all stored notifications).
2018-08-13 16:45:57 +01:00
Rebecca Law
f965322f25 Fixes to the delete letter notifications.
If there are no files to delete we won't get an excpetion.
Wrap the delete file in a try/except to avoid stopping the entire task.
Fix the missing slash for the file name.
2018-08-13 14:09:51 +01:00
Rebecca Law
2a7f3faec9 Merge branch 'master' into fix-delete-s3 2018-08-13 11:44:44 +01:00
Rebecca Law
eb2c878edd Fix bug with deleting the S3 file.
Removed the duplicate method.
2018-08-13 11:33:19 +01:00
Rebecca Law
ebb43082d5 Deal with letters that have not been sent, this can be test letters or letters in tech-failure. 2018-08-10 16:22:25 +01:00
Rebecca Law
f844a39ea6 Move the condition outside the method, remove the notification_type variable. 2018-08-10 13:11:23 +01:00
Rebecca Law
d0e9ab4972 If the notifications that are being deleted are letters then we need to delete the letter from s3 as well. 2018-08-08 16:20:25 +01:00
Rebecca Law
517608dff5 Update method to delete notifications that are a week old, to look at the days of retention set for the service.
If the service does not have the days of retention set, then use 7 days.
Added a method to get days of retention for a service and notificaiton type
2018-08-06 13:51:54 +01:00
Rebecca Law
019c6a4e3a Revert "Purge notifications for configured days of retention" 2018-08-03 14:35:36 +01:00
Rebecca Law
78f8c17fe8 Merge pull request #1965 from alphagov/check-data-retention-before-purging-notifications
Purge notifications for configured days of retention
2018-08-03 13:49:49 +01:00
Pea Tyczynska
d8d94b431f Find email_branding by name 2018-07-31 17:58:19 +01:00
Rebecca Law
67639bf49c Merge branch 'master' into check-data-retention-before-purging-notifications 2018-07-31 11:40:58 +01:00
Rebecca Law
ff2334937c Last night we had some letter notifications that were in permanent failure and had page count = 0.
This causes the create-nighlty-billing task to fail.
This will make sure that 0 is returned in this case.
2018-07-31 11:04:48 +01:00
Pea Tyczynska
ca2b350a99 Remove references to monthly_billing table from api 2018-07-30 11:07:42 +01:00
Leo Hemsted
a826f6e924 make rebuild-ft-billing-for-day remove unused rows
if the billing data was incorrect and needs to be rebuilt, we should
remove old rows. Previously we were only upserting new rows, but old
no-longer-relevant rows were staying in the database. This commit
makes the flask command remove *all* rows for that service and day,
before inserting the rows from the original notification data after.

This commit doesn't change the existing nightly task, nor does it
change the upsert that happens upon viewing the usage page. In normal
usage, there should never be a case where the number of billable units
for a rate decreases. It should only ever increase, thus, never need to
be deleted
2018-07-26 19:10:30 +01:00
Pea (Malgorzata Tyczynska)
1b21a12b83 Merge pull request #1971 from alphagov/add_task_to_send_complaints_on
Add and call task to send complaints on to service callback APIs
2018-07-24 15:07:09 +01:00
Rebecca Law
0675f09afb Need to be able to query the notification statistics for the right number of days.
If the request is for the big numbers on the activity page, then we need to use the number right number of days.
Added an end point to get the data retention for the service and notification type, which is needed on the activity page to say how long the report is available for.
2018-07-23 09:56:04 +01:00
Rebecca Law
dae29a1b61 Update the query for the notifications activity page to return the data for the days of retention if set. 2018-07-23 09:56:04 +01:00
Rebecca Law
9674005c71 Updated the function to delete notifications over a week old to look at ServiceDataRetention to only delete notifications specified number of days for the notification type. 2018-07-23 09:56:03 +01:00
Pea Tyczynska
812f4d20dd Send complaints on to service callback APIs using an async task 2018-07-19 16:59:39 +01:00
Katie Smith
b1cfa8942a Add one_off filter when getting all notifications for a service
Added the option to filter by one_off messages to the DAO function
`get_notifications_for_service`. Previously, one-off notifications
were not returned - this has changed so that the default is for
one-off notifications to be returned. Also simplified the `include_jobs`
filter for this function.

The DAO function gets used in 3 places - for the V1 and V2 API endpoints,
which will now start to return one-off messages. It also gets used by
the admin app which needs to pass in `include_one_off=False` to the
`get_all_notifications_for_service` where we don't want one-off
notifications to show, such as the API message log page.
2018-07-18 15:08:06 +01:00
Pea Tyczynska
86978c225a Filter 'get_service_callback_api_for_service' to only get status updates
Also rename it to 'get_service_delivery_status_callback_api_for_service'
2018-07-18 11:36:39 +01:00
Rebecca Law
9c99e45008 Merge pull request #1945 from alphagov/flexible-data-retention
Flexible data retention
2018-07-17 16:03:18 +01:00
Pea (Malgorzata Tyczynska)
48d1e816ef Merge pull request #1943 from alphagov/admin_platform_users_search
Add data endpoint for finding users by full or partial email
2018-07-16 15:47:23 +01:00
Pea Tyczynska
a69dee5e6d Move code that escapes special chars to helper function and use it
in query get_users_by_partial_email
2018-07-13 15:47:21 +01:00
Pea Tyczynska
782a8ab9e7 Add data endpoint for finding users by full or partial email 2018-07-13 15:47:20 +01:00
Rebecca Law
ab695d24d5 Added new endpoints to get service data retention for a service. 2018-07-13 15:18:27 +01:00
Rebecca Law
e2a1dfeb31 New dao and endpoints to create and update service data retention. 2018-07-11 17:02:49 +01:00
Leo Hemsted
d88003d143 Merge pull request #1940 from alphagov/dashboard-day-limit
Sort out the dashboard day limits
2018-07-11 16:07:31 +01:00
Katie Smith
0cb9e335b9 Delete old platform-stats route
We no longer need the `/platform-stats` route in the service blueprint,
because admin is using the new `/platform-stats` route in the platform stats
blueprint instead.
2018-07-10 14:59:24 +01:00
Katie Smith
c57f33a02d Use real letter_rates in tests
Added the letter_rate table to the list of tables which does not get
deleted after each test run and changed the tests to use the real letter
rates.

Also removed the letter rate DAO since this was only being used in
tests, so was no longer needed.
2018-07-10 11:31:18 +01:00
Leo Hemsted
7fda32b7cf dashboard functions should return data for 8 days, not 7
really, it'll be somewhere btween 7 and 8 depending on what time of day
you request it at. But if today is monday, then seven days ago is last
tuesday - but we should return data for last monday as well so that
users see a full week's worth of data

also update/clarify the tests to make sure this is being honored for
all the different widgets on the dashboard
2018-07-09 12:03:53 +01:00
Leo Hemsted
6705f9d209 Merge pull request #1933 from alphagov/delete-service-stuff
Delete service apis (callback + inbound)
2018-07-06 15:19:01 +01:00
Leo Hemsted
32415b3b14 add delete functions for inbound and callback api objects
Both service api tasks work fine if the object is unexpectedly deleted
halfway through - they both check to see if the api details are still
in the DB before trying to send the request.
2018-07-05 11:09:17 +01:00
Katie Smith
7015e240e0 Merge pull request #1929 from alphagov/paginate-complaints
Paginate complaints
2018-07-04 15:34:46 +01:00
Leo Hemsted
1589bb58bb make sure test keys are filtered out of stats 2018-07-03 14:58:32 +01:00
Leo Hemsted
089056ef72 add tests for new statistics and fact notification status functions 2018-07-03 14:58:32 +01:00
Leo Hemsted
75fdb0290f remove old monthly stats function 2018-07-03 14:57:19 +01:00
Leo Hemsted
2463cd8fb5 move monthly notification stats from notification history to ft tables
it now uses the ft_notification_status table - nice and quick. if you
ask for the current tax year it'll top up the data with numbers from
the notification table.

the data is in exactly the same shape as the old endpoint - no changes
to the admin app are necessary
2018-07-03 14:56:31 +01:00
Katie Smith
3398e4cf93 Paginate the function to get all complaints
The DAO function to get all complaints has been paginated and the
endpoint which uses it has been updated to take a page number as a query
parameter.
2018-07-03 11:11:49 +01:00
Katie Smith
4228b77ebb Fix aggregate stats dao function to use start_date
Now using start_date to determine which table to query.
2018-07-03 09:26:02 +01:00