Commit Graph

3113 Commits

Author SHA1 Message Date
Leo Hemsted
d3449c37da Merge pull request #2593 from alphagov/split-up-nightly-tasks
split up reporting tasks in to separate tasks per day
2019-08-21 15:58:17 +01:00
Rebecca Law
d40cbf8d74 Merge pull request #2594 from alphagov/update-alert-for-letters-in-created
Update alert for letters in created status
2019-08-20 13:49:27 +01:00
Leo Hemsted
8f13697cf1 Revert "trigger nightly delete tasks from the create notification status task"
This reverts commit 58f24a0a83.
2019-08-19 16:06:25 +01:00
Leo Hemsted
36dd750637 split up reporting tasks in to separate tasks per day
to try and speed up overall time by parallelising
2019-08-19 16:06:25 +01:00
Pea (Malgorzata Tyczynska)
7e43b5bbc3 Merge pull request #2587 from alphagov/find_services_by_partial_name
Find services by partial name
2019-08-16 11:21:19 +01:00
Rebecca Law
f097abe82b Change the query to get the notifications for the check_templated_letter_state.
Now looking at the updated_at date, we are getting the alert if the notification was created_at:17:29 updated to created status at 17:30, so the letter is in the next days bucket.

Not sure if I want to make this change, there isn't an index on updated_at, so the query might be slow.
2019-08-16 10:37:51 +01:00
Leo Hemsted
92d78956be Merge pull request #2592 from alphagov/reporting-worker
Add reporting worker
2019-08-15 17:22:27 +01:00
Leo Hemsted
e5c76ffda7 reduce days to process from 10 to 4
to try and speed it up temporarily.
2019-08-15 17:06:38 +01:00
Leo Hemsted
3a0bf2b23e Add reporting worker
also remove references to unused statistics queue
2019-08-15 16:42:15 +01:00
Leo Hemsted
689d76d820 Merge pull request #2590 from alphagov/delete-tasks
trigger nightly delete tasks from the create notification status task
2019-08-15 15:39:00 +01:00
Leo Hemsted
58f24a0a83 trigger nightly delete tasks from the create notification status task
the nightly tasks need to run after the create nightly notification
status task - so that test notifications are still there to record
stats for, and to stop the risk of deleting notificaitons part-way
through recording stats for them.
2019-08-14 18:04:45 +01:00
Rebecca Law
55dc7184cc Add organisation_id to Service.
This is the second commit in the series to add organisation_id to Service.
- Data migration to update services.organisation_id from data in organisation_to_service
 (The rollback will lose any updates to organisation unless the script is updated to set organistion_to_service from service.organisation_id )
- Update Service.organisation relationship to a ForeignKey relationship to Organisation.
- Update Organisation.services to a backref relationship to Service.
2019-08-13 17:25:30 +01:00
Pea Tyczynska
52a1316a7e Find services by partial name 2019-08-13 17:20:37 +01:00
Rebecca Law
515d6602c1 Merge pull request #2583 from alphagov/add-org-id-to-services
Add org id to services
2019-08-13 11:05:29 +01:00
Rebecca Law
ae1bc54f9e Update NotificationTechnicalFailureException
- Change the NotificationTechnicalFailureException so that it only inherits from Exception.
- The notify_celery task should create the logging message on failure.
- Fix unit tests
- Remove named parameter when raising exception.
2019-08-12 16:51:39 +01:00
Rebecca Law
12f305bf86 - Migration to add Service.organisation_id
- Add oranisation_id to Service data model.
- Update methods to create service and associate service to organisation to set the organisation_id on the Service.
- Create the missing test, if the service user email matches a domain for an organisation then associate the service to the organisation and inherit crown and organisation_type from the organisation.
2019-08-12 16:03:02 +01:00
Leo Hemsted
9d8e8747d1 Merge pull request #2582 from alphagov/punycode
punycode encode emails before sending
2019-08-12 15:16:44 +01:00
Leo Hemsted
09d6c60ff1 punycode encode emails before sending
amazon SES only accepts domains encoded in punycode, an encoding that
converts utf-8 into an ascii encoding that browsers and mailservers
recognise.

We currently just send through emails as we store them (in full
unicode), which means any rogue characters break SES and cause us to
put the email in technical-failure. Most of these appear to be typos
and rogue control characters, but there is a small chance that it could
be a real domain (eg https://🅂𝖍𝐤ₛᵖ𝒓.ⓜ𝕠𝒃𝓲/🆆🆃🅵/).

We should encode to and reply-to-address emails as punycode to make
sure that they can always be sent. The chance that anyone actually uses
a unicode domain name for their email is probably pretty low, but we
should allow it for completeness.
2019-08-12 13:53:22 +01:00
Katie Smith
355fb07eb2 Revert "Change email status to permanent-failure if SES raises InvalidParameterValue"
This reverts commit 51716fbaf8.

Instead of relying on catching SES errors we will convert all emails to
punycode before sending instead.
2019-08-12 13:51:24 +01:00
Katie Smith
3b670a0dbe Delete unused delivery blueprint 2019-08-12 10:54:11 +01:00
Katie Smith
51716fbaf8 Change email status to permanent-failure if SES raises InvalidParameterValue
If SES raised an `InvalidParameterValue` error (because an email address
was wrong) we were logging an exception and setting the email status to
`technical-failure`. We now set it to `permanent-failure` instead and
change the log level to `info` - setting it to `permanent-failure` means
that people will know not to retry the message.
2019-08-12 10:24:59 +01:00
Katie Smith
e449e234db Retry deliver_sms task immediately if sending fails
If the `deliver_sms` catches an exception when trying to send an SMS, we
want the first retry to happen immediately (because we will have
switched providers), then every retry after that to happen at the
standard intervals.
2019-08-08 09:34:38 +01:00
Rebecca Law
16fd12be6a If a job was created 7 days ago but scheduled to send 2 days later, the csv file will be deleted from S3 two days early.
For example if a file is uploaded on Aug 1, but scheduled for Aug 3, the csv file will be deleted 2 days before the notifications.
This will cause an error for on the jobs pages if the download report link is clicked.
2019-08-07 16:54:04 +01:00
Leo Hemsted
2c2eb2abad handle 405s
Handle werkzeug http errors separately. Also, improve the generic
`Exception` handler to be more flexible when exceptions don't look
like http errors.

note: because they're 405s the route isn't matched, so the v2 error
handlers don't trip properly. this might be an issue because the
internal endpoints expect a different return format. Might have to
think about how to handle this.
2019-07-26 13:26:20 +01:00
Pea (Malgorzata Tyczynska)
359016de6a Merge pull request #2565 from alphagov/migrate_org_types
Migrate org types
2019-07-26 10:51:42 +01:00
Pea Tyczynska
38bb2c1cf6 Simpify org types migration script and introduce foreign key
Don't transform org types on prod
2019-07-24 17:55:56 +01:00
Katie Smith
b4a6aecbb9 Add endpoint to return monthly notification stats per service
This will be used by notifications-admin to create a CSV report.
2019-07-23 10:00:50 +01:00
Katie Smith
81de71dc1f Add dao function that gets the required data 2019-07-23 10:00:50 +01:00
Pea Tyczynska
5b256fa64e Migrate all orgs and services onto new organisation types
Remove all mentions of generic 'nhs' organisation type.
2019-07-22 14:58:41 +01:00
Pea (Malgorzata Tyczynska)
da61bb3b06 Merge pull request #2557 from alphagov/more_org_types
Introduce new organisation types and organisation_types db table
2019-07-22 14:55:30 +01:00
Pea (Malgorzata Tyczynska)
c0966ad90d Merge pull request #2563 from alphagov/change_content_cancel_letter_job
Change content of error messages when letter job can't be cancelled.
2019-07-22 13:28:11 +01:00
Pea (Malgorzata Tyczynska)
957cf46748 Change apostrophes to prettier ones
Co-Authored-By: Chris Hill-Scott <me@quis.cc>
2019-07-22 10:48:34 +01:00
Chris Hill-Scott
da1219fadf Let services have no default letter contact blocks
If they delete the default we’ll treat ‘Blank’ as the default on the
frontend.
2019-07-22 10:40:18 +01:00
Chris Hill-Scott
dd225fbc6e Dissociate letter contact blocks before archiving
Otherwise you can’t archive it.
2019-07-19 15:40:28 +01:00
Pea Tyczynska
aa88252cf7 Drop making crown not nullable on downgrade as it would fail anyway 2019-07-18 17:05:24 +01:00
Pea Tyczynska
d89ef0594f nhs_central is crown org and also we do not update org type while
updating service anymore.
2019-07-18 17:05:24 +01:00
Pea Tyczynska
e4cb56b5d3 Emergency service instead of emergency services 2019-07-18 17:05:23 +01:00
Pea Tyczynska
8a3ac8400f Update service type and crown when service is added to organisation 2019-07-18 17:05:23 +01:00
Pea Tyczynska
9dfc550f46 Add org types table 2019-07-18 17:05:22 +01:00
Rebecca Law
996dcdd88c Increase the number of days we rebuild the tables for 2019-07-18 16:45:27 +01:00
Pea Tyczynska
e9de84beb0 Change content of error messages when letter job can't be cancelled.
This content was first done without help of content designer so
now it is improved with the help of one. New content is more
in line with the content in the admin app and more human-readable
and following GOV.UK standard.
2019-07-18 16:38:08 +01:00
Rebecca Law
ed611f982c We found a problem with the report tasks that populate the fact tables (or statistic tables). It is possible that the notification status can change for notifications after 4 days.
This PR updates those queries to look in either Notification or NotificationHistory. Since the data does not exist in both tables at the same time we can do with and not worry about the data retention.
The query will iterate over each service, then each notification type and query the data if no results then try the history table.
2019-07-18 15:29:54 +01:00
Katie Smith
1671221642 Set org_type of a service when adding/updating org org type
The organisation_type of a service should match the organisation_type of
the service's organisation (if there is one). This changes
dao_update_organisation and dao_add_service_to_organisation to set the
organisation_type of any services when adding / updating an org.
2019-07-12 16:43:37 +01:00
Leo Hemsted
a0c0dec429 add tests
include dynamically loading the template contents from the alembic upgrade file itself
2019-07-12 15:14:13 +01:00
Chris Hill-Scott
3e11d2492a Merge pull request #2552 from alphagov/create-update-org-extra-fields
Add extra fields to create/update organisation
2019-07-12 15:03:56 +01:00
Chris Hill-Scott
4a2e906dfb Merge pull request #2540 from alphagov/remove-old-fields-from-organisations-and-services
Remove old fields from organisations and services
2019-07-10 11:02:26 +01:00
Pea (Malgorzata Tyczynska)
e3fdfff29b Merge pull request #2543 from alphagov/cancel-letter-job
Cancel letter job
2019-07-04 14:25:17 +01:00
Pea Tyczynska
8588d00657 Move tests closer to the code:
test can_letter_job_be_cancelled closer to the code
test dao_cancel_letter_job closer to the code
Mock out calls in cancel_letter_job to test just that method
2019-07-04 14:12:11 +01:00
Chris Hill-Scott
a4ae534f87 Add extra fields to create/update organisation 2019-07-03 16:46:01 +01:00
Katie Smith
1ebd147ad0 Merge pull request #2551 from alphagov/fix-0-billable-units
Fix 0 billable units on provider callback
2019-07-02 09:12:55 +01:00