We've updated the current service callbacks to add callback types
in #1964, but since the table is versioned we also need to add a
type to the history records.
Even though they're not used anywhere at the moment this might make
it easier to restore from a history callback record in the future.
We now support letters of up to 5 sheets long, so we need to store the
rates for 4 and 5 sheet letters (both crown and non-crown) in the
`letter_rates` table.
We have a few old jobs which don’t have a `processing_started` date.
This means that they always sort to the top of the jobs list in admin,
no matter how old they are. We think this is due to an old bug where
jobs would not be updated if a deploy was in progress.
This commit backfills the `processing_started` data for these jobs,
which will be roughly accurate. Complete accuracy is not the goal;
having these jobs not sort to the top of the list is.
This will affect 5 jobs across 3 services on production:
```sql
select service_id, job_status, created_at, updated_at, processing_started, processing_finished, notification_count, notifications_sent, notifications_delivered, notifications_failed from jobs where processing_started is null and job_status = 'in progress';
```
```
service_id | job_status | created_at | updated_at | processing_started | processing_finished | notification_count | notifications_sent | notifications_delivered | notifications_failed
--------------------------------------+-------------+----------------------------+----------------------------+--------------------+---------------------+--------------------+--------------------+-------------------------+----------------------
d47e5a1b-a04b-4398-8935-c8a266ce1d44 | in progress | 2017-09-29 13:49:41.512356 | 2017-10-01 02:01:05.281162 | | | 10615 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-09-29 09:25:39.802623 | 2017-09-29 16:01:02.154291 | | | 10240 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-09-29 09:31:52.455919 | 2017-09-29 16:01:01.990054 | | | 9930 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-08-22 08:15:39.125999 | 2017-08-22 16:01:07.758805 | | | 6967 | 0 | 0 | 0
95316ff0-e555-462d-a6e7-95d26fbfd091 | in progress | 2016-05-27 14:44:18.114564 | 2016-06-13 00:18:14.542795 | | | 2742 | 2238 | 525 | 1713
(5 rows)
```
We want to add the option for a user to have a more restricted view of
Notify.
This should be opt in/out on a per service basis, because it will be
appropriate only for a minority of teams. It shouldn’t be the default
because most teams (those without a lot of users or those only using the
API) won’t need it.
So this commit adds it as a new service permission, which means we can
roll it out:
- in prototype form initially
- then as an opt-in service setting
This is going to be used for for the document download citizen landing
page, a service will add a contact link
e.g. https://customerservicecontactnumber.uk/dwp/ which will allow the
user to contact the sending department if there is an error or any
issues with the download.
* Added the contact link to the model
* Added db migration script to add the column to the database
If a someone gets an email from one of our services and then complain about it (mark as spam or otherwise), we get a callback from SES.
The service needs to know about these complaints so they can remove that email from their mailing list.
Added created_at and updated_at to the ft_notification_status table in order
to make it easier to track down any potential issues with the data.
Also updated the command to populate the data to take created_at and
updated_at into account and to simplify the command. This can all be done in
the same commit since the table is not being used anywhere yet and can
only be populated manually.
`created_at` was added previously and made nullable temporarily. This
commit now populates the column, ensures that it will always have a
value, and makes `created_at` non-nullable.
Added `created_at` and `updated_at` to the `ft_billing` table - having these
columns makes it easier to track down issues with the data in this table.
`created_at` is nullable initially, but will be changed to non-nullable
once the column is populated and the DAO etc. have been updated.
Added the migration for the `ft_notification_status` table which will be
used to look up the counts for notifications by various different
fields.
The `job_id` column is non-nullable because we want to include it in the
composite primary key. For notifications which weren't part of a job,
the `job_id` will be set to `0000-00000-0000-00000` when populating the
data.
Changed the name of the `is_active` columns to `archived` in these
tables:
* `service_email_reply_to`
* `service_sms_senders`
* `service_letter_contacts`
These columns were not being used yet, but because we describe Templates as
'archived', this keeps the names consistent.
The default value of these columns now needs to be False, not True.
Added a new boolean column, `is_active` to these tables
* service_email_reply_to
* service_sms_senders
* service_letter_contacts
This has a database default of True in order to backfill the data, but
this default will be replaced with a model default later.
As it turns out the SNS topic is trigger more that once when a file is placed in S3, this is caused by a bug in the s3ftp software used to mount the S3 bucket to the FTP server.
S3ftp performs the create file operation more than once. This is resulting in duplicate counts of DailySortedLetter (the counts of how many letters marked as sorted or unsorted, which affect how much the provider will charge Notify for the letter).
This PR adds a new column to DailySortedLetter called file_name. A new unique constraint on billing_day + file_name is added. Each time we write a row to the table the counts are over written rather than aggregated.
I am aware that this PR is not backwards compatiable. However, since the code is typically triggered once a day around 13:00 then it is very unlikely an exception will occur during the deploy. Also a complete migration of the data based on all our response files on S3 will be performed soon, meaning the data will be corrected. Also if an exception does occur it is after the updates to notification status has already occurred.
We need to deal with this, it's ok when updating a notification status from delivered to delivered. But the DailySortedLetter counts are being doubled.
Adding the file_name to the table as a unique key to get around this issue. It will mean we have multiple rows for each billing_day, but that's ok we can aggregate that.
This will also give us a way to see which file created which count.
Added the following new notification statuses:
* pending_virus_check
* virus_scan_failed
If we decide to remove these statuses in future, we will need to replace
them with a different status in the notifications and
Notification_history tables where they are referenced, so
pending-virus-check will be replaced with sending, and virus-scan-failed
will be replaced with permanent-failure.
