Commit Graph

173 Commits

Author SHA1 Message Date
Chris Hill-Scott
5fe0fafadf Archive, don’t delete contact lists
So we keep a record of who first uploaded a list it’s better to archive
a list than completely delete it.

The list in the database doesn’t contain any recipient info so this
isn’t a change to what data we’re retaining.

This means updating the endpoints that get contact lists to exclude ones
that are archived.
2020-03-27 09:51:54 +00:00
Chris Hill-Scott
b50dbab8fd Add an endpoint to delete a contact list
This was one of things we de-scoped when we first shipped this feature.

In order to safely delete a list, we first need to make sure any jobs
aren’t referencing it.
2020-03-26 17:42:38 +00:00
Rebecca Law
f04210ba7d Refactor delete_notifications_older_than_retention_by_type to use a new strategy.
The insert_notification_history_delete_notifications function uses a temp table to store the data to insert and delete. This will save extra queries while performing the insert and delete operations.
The function is written in such a way that if the task is stop while processing when it's started up again it will just pick up where it left off.

I've made a decision to delete all test data in one query, I don't anticipate a problem with that.

The performance of this might also be better than last nights test because we are inserting everything we need for the NotificationHistory insert, so we don't need the join to Notifications to perform the insert.
2020-03-24 12:21:28 +00:00
Rebecca Law
654e6fc657 New table and endpoints for service contact lists.
- Table to store meta data for the emergency contact list for a service.
- Endpoint for fetching contact lists for service
- Endpoint for saving contact list for service.

The list will be stored in S3. The service will then be able to send emergency announcements to staff.
2020-03-13 12:11:16 +00:00
Rebecca Law
c91f37ff4c Change the updates to only look at today, and not yesterday. 2020-02-26 17:38:20 +00:00
Rebecca Law
18f272dc2b Add queries to handle returning usage for all services associated to a given organisation. 2020-02-24 11:28:42 +00:00
Leo Hemsted
1694395b17 record document count when processing api notifications
if someone doesn't send any documents, set the value to None. If it's
not specified, it defaults to None anyway.
2020-02-13 12:43:06 +00:00
Pea Tyczynska
0eed4c99a7 Add email_access_valdiated_at field to user table, populate it
and update it when users have to use their email to interact with
Notify service.

Initial population:
If user has email_auth, set last_validated_at to logged_in_at.
If user has sms_auth,  set it to created_at.

Then:
Update email_access_valdiated_at date when:
- user with email_auth logs in
- new user is created
- user resets password when logged out, meaning we send them an
email with a link they have to click to reset their password.
2020-01-30 14:51:54 +00:00
Leo Hemsted
a16f79896b refer to file rather than document
Karl Approved™
2020-01-20 13:44:52 +00:00
Rebecca Law
aabaa4a971 Added joins to template, job and user for returned letter query.
Added unit tests
Comleted endpoint to get returned letter details
2019-12-27 10:27:59 +00:00
Rebecca Law
5482c03bca [WIP] 2019-12-27 10:27:59 +00:00
Rebecca Law
d330025447 Changed reported_at to a date and included audit columns. 2019-12-12 14:11:54 +00:00
Rebecca Law
140cb65554 Return report_at date as a date not datetime. I haven't converted this date from UTC to BST because we always upload the returned-letters during business hours. 2019-12-11 14:41:35 +00:00
Rebecca Law
c8364b4dc4 Add endpoint to return the summary data for returned letters.
Returning the count of letters that are returned for each report date.
2019-12-10 16:21:55 +00:00
Rebecca Law
40a0c62926 New endpoint to return a summary of returned letters for the service. 2019-12-09 17:27:18 +00:00
Leo Hemsted
dd57468147 remove notification_type and service from create_ft_billing
they can both be inferred from the template, and specifying them just
leads to unnecessary risk of errors
2019-12-03 17:02:58 +00:00
Leo Hemsted
ed5a52fe0d make create_ft_billing ensure data is correct
that is, the template must belong to the named service, and the
template's template_type must match a provided notification_type
2019-12-03 16:05:48 +00:00
Leo Hemsted
d83827579e make ft billing nightly task only look at one table
follows same logic as the create_nightly_notification_status task, see previous commit
for logic
2019-12-03 14:19:13 +00:00
Rebecca Law
db5a50c5a7 Adding a scheduled task to processing missing rows from job
Sometimes a job finishes but has missed a row in the middle. It is a mystery why this is happening, it could be that the task to save the notifications has been dropped.
So until we solve the missing let's find missing rows and process them.

A new scheduled task has been added to find any "finished" jobs that do not have enough notifications created. If there are missing notifications the job processes those rows for the job.
Adding the new task to beat schedule will be done in the next commit.

A unique key constraint has been added to Notifications to ensure that the row is not added twice. Any index or constraint can affect performance, but this unique constraint should not affect it enough for us to notice.
2019-11-06 10:49:46 +00:00
Katie Smith
97b807a99d Add test function to create service whitelist
This will be used to stop us needing to call the `sample_service_whitelist` fixture as
a function.
2019-10-31 15:04:23 +00:00
Katie Smith
62735ea125 Remove sample_notification_with_api_key fixture
This was only used once and was not Pytest 4 compatible.
2019-09-18 11:04:24 +01:00
Rebecca Law
e64ae321cf The sheet count was not calculated properly (it should be billable_units/notifications_sent).
And it turns out the sheet count is not required for the report. This PR takes out the columns to resolve the group by error.
2019-09-03 13:16:08 +01:00
Rebecca Law
cb7fff6100 Add endpoint to return structured data 2019-08-30 12:17:59 +01:00
Rebecca Law
236d111c22 Update queries for data model change (removal of organisation_to_service)
New query to get letter breakdown.
2019-08-30 12:17:59 +01:00
Pea Tyczynska
9dfc550f46 Add org types table 2019-07-18 17:05:22 +01:00
Chris Hill-Scott
4a2e906dfb Merge pull request #2540 from alphagov/remove-old-fields-from-organisations-and-services
Remove old fields from organisations and services
2019-07-10 11:02:26 +01:00
Chris Hill-Scott
0955403a3b Slim down the /organisations response
At the moment this response returns a list of service IDs for hundreds
of organisations.

The admin app doesn’t use this information, but having to wait for it to
be serialized and sent across the network slows it down all the same.
2019-06-18 10:48:41 +01:00
Rebecca Law
1f9aade749 - Adding test where post_notification can't send to a member outside of the team or whitelist.
- Updating some tests to be pytest4 compliant.
2019-06-11 16:45:35 +01:00
Katie Smith
844f22bcd0 Remove unused fixture and add test function to create permissions 2019-06-05 14:54:52 +01:00
Leo Hemsted
67f171f2f8 refactor tests
re-order notification dao delete notifications test to move "fixtures"
to the top of the file

changed create_service_data_retention to take an ORM object, not an id.
brings it in line with other db.py test functions
2019-06-03 17:27:08 +01:00
Rebecca Law
c23ae15f32 Remove insert to NotificationHistory
Fix all test failures
2019-05-31 16:52:22 +01:00
Rebecca Law
43334d63f3 Stop updating NotificationHistory
Doing my bit to remove imports of fixtures.
2019-05-15 10:58:39 +01:00
Pea Tyczynska
689ff57c32 Add organisation type to live services report 2019-05-03 15:09:54 +01:00
Rebecca Law
55a65a22a9 Added test for go_live_user and go_live_at in the response for get_service_by_id 2019-04-17 13:20:44 +01:00
Chris Hill-Scott
ee966668bd Remove domain columns from branding table
This relationship is via the `Organisation` now; we don’t use this
column to fudge a relationship based on the user’s email address and the
matching something in these columns.
2019-04-12 15:53:17 +01:00
Chris Hill-Scott
6bfd999de2 Add method to count of live services and orgs
So that we don’t have to update the home page manually any more.
2019-04-11 13:54:43 +01:00
Leo Hemsted
28bff28786 remove letter rates between tests
we only use them in ft_billing so no reason not to delete them. makes
the tests read better as it's obvious how they work too.
2019-04-03 15:34:02 +01:00
Leo Hemsted
6f41f6c7d7 use db models instead of tuples when referring to rate objects
makes it less confusing
2019-04-03 13:08:53 +01:00
Katie Smith
31ddd36e2c Update invited user tests to take folder_permissions into account 2019-03-22 09:30:39 +00:00
Chris Hill-Scott
cf2c032c90 Merge pull request #2368 from alphagov/richer-orgs
Use ‘organisations‘ table to store info about organisations (not YAML)
2019-03-08 14:57:21 +00:00
Chris Hill-Scott
c0fb9267bd Automatically associate new service with an org
This is the same thing we do in the admin app at the moment with YAML:
2f4e933b65/app/utils.py (L556-L562)
2019-03-08 13:30:42 +00:00
Rebecca Law
1625371106 Merge pull request #2381 from alphagov/inbound-sms-retention
Inbound sms now deletes according to data retention
2019-03-08 10:58:01 +00:00
Leo Hemsted
e7b8f83fc8 make inbound sms honour data retention
code inspired by the delete notification code, but with some clean up
since we don't deal with different types etc, and only need to run the
query for services with inbound numbers

also, update tests.app.db.create_inbound_sms to create inbound numbers
and assign them to services to ensure the test db is always accurate
and reflects real world usage
2019-02-27 15:53:55 +00:00
Pea Tyczynska
211d3741ba Send confirmation emails to users when team manager edits their
email address  or mobile number.
2019-02-26 16:30:29 +00:00
Pea Tyczynska
8ef581de69 Remove service.postage and choose_postage permission from the code and tests 2019-02-11 18:20:34 +00:00
Pea Tyczynska
5b9e6c2086 Set postage on basis of postage argument or template postage 2019-02-06 13:23:09 +00:00
Rebecca Law
e030c2be88 Removing platform_default as a concept. No service actually wants to send letters with the default hm-government logo so we are going to remove it as a constraint.
However, until we can create a letter without a logo, we will still default to hm-government, because the dvla_organisation is set on the service.
This does simplify the code.
Also removed the inserts to letter_branding in the data migration file, because we can deploy this before the rest of the work is finished. But we will need to do it later.
2019-01-25 15:03:01 +00:00
Rebecca Law
f11aa55e0b Adding new data models for letter branding. 2019-01-22 17:27:00 +00:00
Pea Tyczynska
eea324a19d Add flag on create_service to decide whether we should also check if service exists 2019-01-03 10:57:29 +00:00
Pea Tyczynska
0bcf13d85c sample_api_key uses create_service instead of sample_service 2019-01-03 10:52:14 +00:00