The `@version_class` decorator looks at every dirty (modified) model in
the session to work out which new history models to create. However, if
there are dirty items in the session, sqlalchemy might flush to the
database, clearing the whole session.
We ran into problems with the archive service function, which is
versioned for api keys, templates and services. When constructing the
TemplateHistory objects, `history_meta.py::create_history` would call
getattr on `Template.folders`, which would make a database call to join
across to the TemplateFolder objects - this would then flush the dirty
Service object from the session before the ServiceHistory object was
created.
To get around this, we eager load the Template.folder object, joining
on to it automatically when the Template is fetched. That way, it
doesn't make a SELECT mid-way through the version decorator, and the
history is preserved.
Note: This relationship is only on Template, not TemplateHistory - so
we're not doing this join every single time we send a message.
a TemplateFolder has a service, a name, and a parent. Parent is a
nullable foreign key pointing to another TemplateFolder instance. We
don't do any checks here for cyclical or otherwise invalid folder
structures so keep your data clean, folks!
Unsurprisingly, a Template can be part of a TemplateFolder - there's a
mapping class (template_folder_map to avoid giving it a dumb name) -
this mapping table shouldn't be interacted with directly - rather, you
should use the `Template.folder` or `TemplateFolder.templates`
relationship.
- pass new, sanitised pdf for sending
- move invalid pdfs to a newly created bucket
- set status fro notifications that failed pdf validation to a new status validation-failed
- adjust existing tests
Added a filename column to the dvla_organisation table and populated it
with the filenames that are currently hard-coded in template-preview.
The filenames for letter logos are going to be stored in the database,
instead of in template-preview.
The FactBilling model and the ft_billing database table have diverged
slightly - this makes some minor changes to the model columns so that
the model matches the table (which appears to be the correct version).
The ft_billing table is currently like this:
Column | Type | Modifiers | Storage | Stats target | Description
--------------------+-----------------------------+-----------+----------+--------------+-------------
bst_date | date | not null | plain | |
template_id | uuid | not null | plain | |
service_id | uuid | not null | plain | |
notification_type | text | not null | extended | |
provider | text | not null | extended | |
rate_multiplier | integer | not null | plain | |
international | boolean | not null | plain | |
rate | numeric | not null | main | |
billable_units | integer | | plain | |
notifications_sent | integer | | plain | |
updated_at | timestamp without time zone | | plain | |
created_at | timestamp without time zone | not null | plain | |
Indexes:
"ft_billing_pkey" PRIMARY KEY, btree (bst_date, template_id, service_id, rate_multiplier, provider, notification_type, international, rate)
"ix_ft_billing_bst_date" btree (bst_date)
"ix_ft_billing_service_id" btree (service_id)
"ix_ft_billing_template_id" btree (template_id)
There are two fun quirks of postgres/sql that we need to work around:
* any `x = y` where x or y is NULL returns NULL, rather than false.
* check constraints accept NULL or true values as good.
so, the check `postage in ('first', 'second')` returns `null` rather
than `false` when postage is null itself. This surprisingly passes the
check constraint. To get around this, we have to add an explicit not
null check as well.
A not valid constraint only checks against new rows, not existing rows.
We can call VALIDATE CONSTRAINT against this new constraint to check
the old rows (which we know are good, having run the command from
74961781). Adding a normal constraint acquires an ACCESS EXCLUSIVE
lock, but validate constraint only needs a SHARE UPDATE EXCLUSIVE lock.
see 9d4b8961 and 0a50993f for more information on marking constraints
as not valid.
To start with this will be an attribute on the service, at the time the notification is created it will look at Service.letter_class to decide what class to use for the letter.
This PR adds Service.letter class as a nullable column.
Updated the create_service and update_service method to default the value to second.
Subsequent PRs will add the check constraint to ensure we only get first or second in the letter_class column and make that column nullable.
This can't be done all at once because it will cause an error if someone inserts or updates a service during the deploy.
Sets the updated_at to current time when the NotificationHistory
is modified (which happens occasionally for example to set status
to returned letter).
The concern about performnace degrading has been thought through. We do not believe there will be an adverse effect since the high volume users do not send off messages.
Adds an API endpoint `/letters/returned` that accepts a list of
notification references and creates a task to update their status.
Adds a new task that uses the list of references to update the status
of notifications to 'returned-letter'.
The update is currently done using a single query and logs the
number of changed records (including notification history records).
This could potentially be done within the `/letters/returned` endpoint,
but creating a job right away allows us to extend this more easily
in the future (e.g. logging missing notifications or adding callbacks).
The job is using the database tasks queue.
We need to update letter notifications with a new status when DVLA
gives us a list of references for returned letters.
This adds the new status to the models and the DB.
DVLA call this 'returned mail', so I'm using it as the status name
since it seems less ambiguous than 'returned'.
Does two things:
1. Revert "Revert "Add unique constraint to email branding domain""
This reverts commit af9cb30ef3.
2. Don’t allow empty string in email branding domain
Columns with multiple `null`s can have a uniqueness constraint. Columns
with multiple empty string values are not considered unique.
This commit:
- removes any duplicate empty string values
- casts empty strings to null string any time these columns are updated
---
Squashed into this single commits because these two things are not
atomic as individual commits.
Brandings with a domain set should be considered canonical. It doesn’t
make sense to have the same domain set on multiple different email
brands – you can’t tell which one to use.
‘GOV.UK’ doesn’t make sense as a type of brand. It only made sense as
a type of branding that a service had.
Since we’ve:
- deprecated the service branding column
- made sure it’s not used as a value in the email branding table
we can remove this value from the table of possible brand types.
There is a requirement for the colour of the banner to be different to the single id colour. In order to accomodate that we are adding two new columns for that data.
These columns will be updated manually and not done with a data migration.
A new column called domain has been added. This will be used as a default domain, if the user that creates the service has a matching domain this email brand will be set for the service.
This PR only adds the new columns to the model and db.
https://www.pivotaltracker.com/story/show/159660295
The `serialize` method of the Notification model now includes the
`created_by_name`. If a notification was sent as a one-off message this
value will now be the name of the person who sent the notification. If
a notification was sent through the API or by a CSV upload we don't
record the id of the sender, so `created_by_name` will be `None`.
This change affects the data that gets returned from these endpoints:
* /v2/notifications/<notification_id>
* /v2/notifications
We want to add the option for a user to have a more restricted view of
Notify.
This should be opt in/out on a per service basis, because it will be
appropriate only for a minority of teams. It shouldn’t be the default
because most teams (those without a lot of users or those only using the
API) won’t need it.
So this commit adds it as a new service permission, which means we can
roll it out:
- in prototype form initially
- then as an opt-in service setting
This is going to be used for for the document download citizen landing
page, a service will add a contact link
e.g. https://customerservicecontactnumber.uk/dwp/ which will allow the
user to contact the sending department if there is an error or any
issues with the download.
* Added the contact link to the model
* Added db migration script to add the column to the database
If a someone gets an email from one of our services and then complain about it (mark as spam or otherwise), we get a callback from SES.
The service needs to know about these complaints so they can remove that email from their mailing list.
Added created_at and updated_at to the ft_notification_status table in order
to make it easier to track down any potential issues with the data.
Also updated the command to populate the data to take created_at and
updated_at into account and to simplify the command. This can all be done in
the same commit since the table is not being used anywhere yet and can
only be populated manually.
`created_at` was added previously and made nullable temporarily. This
commit now populates the column, ensures that it will always have a
value, and makes `created_at` non-nullable.
Added `created_at` and `updated_at` to the `ft_billing` table - having these
columns makes it easier to track down issues with the data in this table.
`created_at` is nullable initially, but will be changed to non-nullable
once the column is populated and the DAO etc. have been updated.
