Commit Graph

879 Commits

Author SHA1 Message Date
Katie Smith
7f918d7c48 Remove relationship to folder when archiving a template
When a template is archived, it should no longer belong to any folder.

If we don’t do this it will make it very hard to delete folders later
(because folders can only be deleted if they have no templates or folders
inside them).

We originally tried to check if the link between a template and folder
should be removed with

`if template.archived and template.folder:`

instead of using `if template.archived:`. However, this caused issues
because checking `template.folder` flushes the session. Since the
session is no longer dirty, the versioning decorator doesn't work as
expected and doesn't create a new row in `TemplateHistory`.
2018-11-08 17:25:45 +00:00
Pea (Malgorzata Tyczynska)
5588ddceb7 Merge pull request #2216 from alphagov/rearrange_nightly_tasks
Rearrange nightly tasks and update ft_notification_status table
2018-11-08 14:30:56 +00:00
Pea Tyczynska
ca2db56b9d Update ft_notification_status now deletes old version of data
instead of overwriting on top of it
2018-11-08 11:52:40 +00:00
Rebecca Law
39198ed67e Using jsonschema for create_template.
Updated jsonschema to Draft7, this allowed a conditional validation on subject, if template_type == 'email' or 'letter' then subject is required.
This version is backward compatible with Draft4.
When creating TempalteRedacted, I've built the dict depending on if the created_by or created_by_id exists.
2018-11-07 16:17:08 +00:00
Alexey Bezhan
36f41c23e1 Always use both folder and service ID when getting template folder
Currently there aren't any permission checks based on folder IDs in
the admin app or the API, so it's possible for a user to modify the
folder ID to perform operations on folders outside their service.

Our usual way to avoid this is to always use service_id filter when
fetching objects from the database.
2018-11-07 13:51:37 +00:00
Pea (Malgorzata Tyczynska)
d46caa184e Merge pull request #2211 from alphagov/fix_notification_statistics_on_dashboard
Fix notification statistics on dashboard
2018-11-06 16:22:00 +00:00
Pea Tyczynska
c146b86643 Adjust attribute key names and data types to work with format_statistics 2018-11-06 13:30:37 +00:00
Pea Tyczynska
c37c399741 Add results for notification stats for last 7 days and today 2018-11-06 11:57:08 +00:00
Leo Hemsted
51a7a0add7 Merge pull request #2199 from alphagov/template-folder-crud
add template folder CRUD
2018-11-01 10:44:48 +00:00
Leo Hemsted
fbe34041d6 add template folder CRUD
* create template folder
* rename template folder
* get list of template folders for service (not nested/presented in any
  particular way)
* delete template folder

Also removed `lazy=dynamic` from the `template_folder.templates`
relationship. lazy=dynamic returns a query object (which you can then
filter further). We just want to return the entire fetched list, at
least for now.
2018-10-31 14:28:16 +00:00
Pea Tyczynska
9015ffef48 Fetch notification statistics for last 7 days frpm FactNotificationStatus 2018-10-30 16:31:57 +00:00
Rebecca Law
8832c6d82f Update billing to ignore failed letters. 2018-10-29 14:02:40 +00:00
Rebecca Law
00f04c33c8 Some minor refactoring.
- Updated notifications_dao.update_notification_status_by_id with an optional parameter to set the sent_by, this will eliminate a separate update to notifcaitons.
- Added the callback url to the log message, that way we can see if it's the same url failing.
- Stop sending the status callbacks for PENDING status.
2018-10-24 11:24:53 +01:00
Leo Hemsted
2ed50e760f Revert "Celery 4" 2018-10-09 13:27:49 +01:00
Leo Hemsted
ba7cf10393 Merge pull request #2131 from alphagov/celery4
Celery 4
2018-10-05 16:03:44 +01:00
Leo Hemsted
6ca2b8277c import exception from botocore
boto (2) is no longer a dependency
2018-10-03 14:11:30 +01:00
Katie Smith
236bbc5f28 Return postage from the monthly-usage endpoint
We were already returning the month, notification_type, billing_units
and rate from the /monthly-usage billing endpoint. This adds in the
postage too so that we can display postage details on the usage page of
admin.
2018-10-03 13:07:46 +01:00
Katie Smith
71d28035dd Update ft_billing with real postage data
* Changed update_fact_billing DAO function to update the table with the
real data for postage instead of hard-coding in 'second'.
* Added a test for the create nightly billing task to test that rows
with different postage are being inserted correctly.
2018-09-28 16:32:18 +01:00
Katie Smith
6727f0e0f5 Update ft_billing DAO functions to use postage
* Updated the 'fetch_billing_data_for_day' DAO function to take postage into
account
* Updated the 'update_fact_billing' DAO function to insert postage for
new rows. When updating rows which are identical apart from the postage, the
original row will be kept. (This behaviour will change once postage is
added to the primary key - at this point, upserting will add a new row.)
* Also changed some fixtures / test set up functions to take postage
into account
2018-09-28 13:52:17 +01:00
Pea Tyczynska
165b65612e Change how rates are filtered by date to equal or greater 2018-09-25 15:10:16 +01:00
Pea Tyczynska
b3191dbc2a Filter letter rates by post_class in get_rate
Also adjust existing tests.
2018-09-25 15:10:15 +01:00
Rebecca Law
5f73dfdbe9 Added constraints to services.postage: not nullable and can only containt "first" or "second" 2018-09-18 17:41:25 +01:00
Rebecca Law
7200ccb452 Remove print 2018-09-18 16:44:07 +01:00
Rebecca Law
1c0892888f Change the name of letter_class to postage, this will match the domain language used in the documentation.
Updated the tests to answer review comments.
2018-09-18 15:22:08 +01:00
Rebecca Law
79815b4f8a In order to support sending letters by first class we need a way to know how the service wants us to send the letter.
To start with this will be an attribute on the service, at the time the notification is created it will look at Service.letter_class to decide what class to use for the letter.

This PR adds Service.letter class as a nullable column.
Updated the create_service and update_service method to default the value to second.

Subsequent PRs will add the check constraint to ensure we only get first or second in the letter_class column and make that column nullable.
This can't be done all at once because it will cause an error if someone inserts or updates a service during the deploy.
2018-09-17 11:27:43 +01:00
Rebecca Law
f1b04193ca In this PR we remove trigger-letter-pdfs-for-day scheduled task and just call collate_letter_pdfs_for_day instead.
There was a datetime bug in the query which resulted in files not being sent to the postal provider.
The trigger-letter-pdfs-for-day task is no longer needed, so rather than fix the query just call collate_letter_pdfs_for_day directly.
Less code is always better.

Deployment considerations: I realized this is strictly not backwards compatible if the scheduled job is in progress and a task is on the queue that no longer exists. This is ok since we will deploy this well before 17:50.
2018-09-12 17:16:34 +01:00
Alexey Bezhan
18ab7f3337 Add updated history count to dao_update_notifications_by_reference
For returned letter updates most notifications won't exist in the
notifications table, so in order to find out whether the reference
matches any known letters we need to check the count of updated
history records.
2018-09-04 16:55:22 +01:00
Chris Hill-Scott
5d5047fb13 Add unique constraint to email branding domains
Does two things:

1. Revert "Revert "Add unique constraint to email branding domain""

This reverts commit af9cb30ef3.

2. Don’t allow empty string in email branding domain

Columns with multiple `null`s can have a uniqueness constraint. Columns
with multiple empty string values are not considered unique.

This commit:
- removes any duplicate empty string values
- casts empty strings to null string any time these columns are updated

---

Squashed into this single commits because these two things are not
atomic as individual commits.
2018-09-04 11:10:46 +01:00
Alexey Bezhan
5f2724c429 Add limit_days argument to notification statistics endpoint
Allows getting notification counts for a given number of days to
support services with custom data retention periods (admin dashboard
page should still display counts for the last 7 days, while the
notifications page displays all stored notifications).
2018-08-13 16:45:57 +01:00
Rebecca Law
f965322f25 Fixes to the delete letter notifications.
If there are no files to delete we won't get an excpetion.
Wrap the delete file in a try/except to avoid stopping the entire task.
Fix the missing slash for the file name.
2018-08-13 14:09:51 +01:00
Rebecca Law
2a7f3faec9 Merge branch 'master' into fix-delete-s3 2018-08-13 11:44:44 +01:00
Rebecca Law
eb2c878edd Fix bug with deleting the S3 file.
Removed the duplicate method.
2018-08-13 11:33:19 +01:00
Rebecca Law
ebb43082d5 Deal with letters that have not been sent, this can be test letters or letters in tech-failure. 2018-08-10 16:22:25 +01:00
Rebecca Law
f844a39ea6 Move the condition outside the method, remove the notification_type variable. 2018-08-10 13:11:23 +01:00
Rebecca Law
d0e9ab4972 If the notifications that are being deleted are letters then we need to delete the letter from s3 as well. 2018-08-08 16:20:25 +01:00
Rebecca Law
517608dff5 Update method to delete notifications that are a week old, to look at the days of retention set for the service.
If the service does not have the days of retention set, then use 7 days.
Added a method to get days of retention for a service and notificaiton type
2018-08-06 13:51:54 +01:00
Rebecca Law
019c6a4e3a Revert "Purge notifications for configured days of retention" 2018-08-03 14:35:36 +01:00
Rebecca Law
78f8c17fe8 Merge pull request #1965 from alphagov/check-data-retention-before-purging-notifications
Purge notifications for configured days of retention
2018-08-03 13:49:49 +01:00
Pea Tyczynska
d8d94b431f Find email_branding by name 2018-07-31 17:58:19 +01:00
Rebecca Law
67639bf49c Merge branch 'master' into check-data-retention-before-purging-notifications 2018-07-31 11:40:58 +01:00
Rebecca Law
ff2334937c Last night we had some letter notifications that were in permanent failure and had page count = 0.
This causes the create-nighlty-billing task to fail.
This will make sure that 0 is returned in this case.
2018-07-31 11:04:48 +01:00
Pea Tyczynska
ca2b350a99 Remove references to monthly_billing table from api 2018-07-30 11:07:42 +01:00
Leo Hemsted
a826f6e924 make rebuild-ft-billing-for-day remove unused rows
if the billing data was incorrect and needs to be rebuilt, we should
remove old rows. Previously we were only upserting new rows, but old
no-longer-relevant rows were staying in the database. This commit
makes the flask command remove *all* rows for that service and day,
before inserting the rows from the original notification data after.

This commit doesn't change the existing nightly task, nor does it
change the upsert that happens upon viewing the usage page. In normal
usage, there should never be a case where the number of billable units
for a rate decreases. It should only ever increase, thus, never need to
be deleted
2018-07-26 19:10:30 +01:00
Pea (Malgorzata Tyczynska)
1b21a12b83 Merge pull request #1971 from alphagov/add_task_to_send_complaints_on
Add and call task to send complaints on to service callback APIs
2018-07-24 15:07:09 +01:00
Rebecca Law
0675f09afb Need to be able to query the notification statistics for the right number of days.
If the request is for the big numbers on the activity page, then we need to use the number right number of days.
Added an end point to get the data retention for the service and notification type, which is needed on the activity page to say how long the report is available for.
2018-07-23 09:56:04 +01:00
Rebecca Law
dae29a1b61 Update the query for the notifications activity page to return the data for the days of retention if set. 2018-07-23 09:56:04 +01:00
Rebecca Law
9674005c71 Updated the function to delete notifications over a week old to look at ServiceDataRetention to only delete notifications specified number of days for the notification type. 2018-07-23 09:56:03 +01:00
Pea Tyczynska
812f4d20dd Send complaints on to service callback APIs using an async task 2018-07-19 16:59:39 +01:00
Katie Smith
b1cfa8942a Add one_off filter when getting all notifications for a service
Added the option to filter by one_off messages to the DAO function
`get_notifications_for_service`. Previously, one-off notifications
were not returned - this has changed so that the default is for
one-off notifications to be returned. Also simplified the `include_jobs`
filter for this function.

The DAO function gets used in 3 places - for the V1 and V2 API endpoints,
which will now start to return one-off messages. It also gets used by
the admin app which needs to pass in `include_one_off=False` to the
`get_all_notifications_for_service` where we don't want one-off
notifications to show, such as the API message log page.
2018-07-18 15:08:06 +01:00
Pea Tyczynska
86978c225a Filter 'get_service_callback_api_for_service' to only get status updates
Also rename it to 'get_service_delivery_status_callback_api_for_service'
2018-07-18 11:36:39 +01:00