* Changed update_fact_billing DAO function to update the table with the
real data for postage instead of hard-coding in 'second'.
* Added a test for the create nightly billing task to test that rows
with different postage are being inserted correctly.
* Updated the 'fetch_billing_data_for_day' DAO function to take postage into
account
* Updated the 'update_fact_billing' DAO function to insert postage for
new rows. When updating rows which are identical apart from the postage, the
original row will be kept. (This behaviour will change once postage is
added to the primary key - at this point, upserting will add a new row.)
* Also changed some fixtures / test set up functions to take postage
into account
Updated the 'migrate-data-to-ft-billing' command to populate the new
postage column of ft_billing. This will be populated with the
postage of the notification for letters, or 'none' for email or sms. We
need to ensure there are no null values in postage so that the postage
column can become part of the primary key later.
Also updated the query to get the right rate letter rate now that we are
updating rates in the letter_rates table.
The FactBilling model and the ft_billing database table have diverged
slightly - this makes some minor changes to the model columns so that
the model matches the table (which appears to be the correct version).
The ft_billing table is currently like this:
Column | Type | Modifiers | Storage | Stats target | Description
--------------------+-----------------------------+-----------+----------+--------------+-------------
bst_date | date | not null | plain | |
template_id | uuid | not null | plain | |
service_id | uuid | not null | plain | |
notification_type | text | not null | extended | |
provider | text | not null | extended | |
rate_multiplier | integer | not null | plain | |
international | boolean | not null | plain | |
rate | numeric | not null | main | |
billable_units | integer | | plain | |
notifications_sent | integer | | plain | |
updated_at | timestamp without time zone | | plain | |
created_at | timestamp without time zone | not null | plain | |
Indexes:
"ft_billing_pkey" PRIMARY KEY, btree (bst_date, template_id, service_id, rate_multiplier, provider, notification_type, international, rate)
"ix_ft_billing_bst_date" btree (bst_date)
"ix_ft_billing_service_id" btree (service_id)
"ix_ft_billing_template_id" btree (template_id)
There are two fun quirks of postgres/sql that we need to work around:
* any `x = y` where x or y is NULL returns NULL, rather than false.
* check constraints accept NULL or true values as good.
so, the check `postage in ('first', 'second')` returns `null` rather
than `false` when postage is null itself. This surprisingly passes the
check constraint. To get around this, we have to add an explicit not
null check as well.
A not valid constraint only checks against new rows, not existing rows.
We can call VALIDATE CONSTRAINT against this new constraint to check
the old rows (which we know are good, having run the command from
74961781). Adding a normal constraint acquires an ACCESS EXCLUSIVE
lock, but validate constraint only needs a SHARE UPDATE EXCLUSIVE lock.
see 9d4b8961 and 0a50993f for more information on marking constraints
as not valid.
sets all letters in notification history (and notifications) to
"second", so that there's no null letters for billing etc. Commits in
ten day chunks (up to ~30k letters)
To start with this will be an attribute on the service, at the time the notification is created it will look at Service.letter_class to decide what class to use for the letter.
This PR adds Service.letter class as a nullable column.
Updated the create_service and update_service method to default the value to second.
Subsequent PRs will add the check constraint to ensure we only get first or second in the letter_class column and make that column nullable.
This can't be done all at once because it will cause an error if someone inserts or updates a service during the deploy.
Sets the updated_at to current time when the NotificationHistory
is modified (which happens occasionally for example to set status
to returned letter).
There was a datetime bug in the query which resulted in files not being sent to the postal provider.
The trigger-letter-pdfs-for-day task is no longer needed, so rather than fix the query just call collate_letter_pdfs_for_day directly.
Less code is always better.
Deployment considerations: I realized this is strictly not backwards compatible if the scheduled job is in progress and a task is on the queue that no longer exists. This is ok since we will deploy this well before 17:50.
The concern about performnace degrading has been thought through. We do not believe there will be an adverse effect since the high volume users do not send off messages.
Adds an API endpoint `/letters/returned` that accepts a list of
notification references and creates a task to update their status.
Adds a new task that uses the list of references to update the status
of notifications to 'returned-letter'.
The update is currently done using a single query and logs the
number of changed records (including notification history records).
This could potentially be done within the `/letters/returned` endpoint,
but creating a job right away allows us to extend this more easily
in the future (e.g. logging missing notifications or adding callbacks).
The job is using the database tasks queue.
For returned letter updates most notifications won't exist in the
notifications table, so in order to find out whether the reference
matches any known letters we need to check the count of updated
history records.
We need to update letter notifications with a new status when DVLA
gives us a list of references for returned letters.
This adds the new status to the models and the DB.
DVLA call this 'returned mail', so I'm using it as the status name
since it seems less ambiguous than 'returned'.
Does two things:
1. Revert "Revert "Add unique constraint to email branding domain""
This reverts commit af9cb30ef3.
2. Don’t allow empty string in email branding domain
Columns with multiple `null`s can have a uniqueness constraint. Columns
with multiple empty string values are not considered unique.
This commit:
- removes any duplicate empty string values
- casts empty strings to null string any time these columns are updated
---
Squashed into this single commits because these two things are not
atomic as individual commits.
Brandings with a domain set should be considered canonical. It doesn’t
make sense to have the same domain set on multiple different email
brands – you can’t tell which one to use.
‘GOV.UK’ doesn’t make sense as a type of brand. It only made sense as
a type of branding that a service had.
Since we’ve:
- deprecated the service branding column
- made sure it’s not used as a value in the email branding table
we can remove this value from the table of possible brand types.
https://www.pivotaltracker.com/story/show/159986276
We are now setting the type of branding on the branding itself, not on
the service.
This commit switches over from looking in the old place (on the service)
to looking in the new place (on the branding).
Notify antivirus, on success, calls the process_virus_scan_passed taks.
Previously, this task would:
* update status to created (or delivered for test keys)
* copy the file from the scan bucket to either the live or test bucket
based on the results
* delete the old file in the scan bucket
We want it to:
* download file from scan bucket
* sanitise PDF using new template-preview functionality
* if sanitise failed, set to new status "validation-failed" and save
the pdf somewhere.
* send new pdf to live/test bucket
* update status to created (or delivered for test keys)
* delete the original file in the scan bucket
This PR does some of that:
* download file from scan bucket
* sanitise PDF using new template-preview functionality
* if sanitise failed, just log.
* send OLD pdf to live/test bucket
* update status to created (or delivered for test keys)
* delete the original file in the scan bucket
So if sanitising fails, we won't fall over and not deliver the letter,
we'll just log a message for now. If sanitise throws an unexpected
error (as opposed to a 400), we'll retry up to fifteen times (the same
as when creating a new letter). I've added the code for using the
sanitised pdf, but it's commented out for now
