Currently we have
- a thing in the database called an ‘organisation’ which we don’t use
- the idea of an organisation which we derive from the user’s email
address and is used to set the default branding for their service and
determine whether they’ve signed the MOU
We should make these two things into one thing, by storing everything
we know about an organisation against that organisation in the database.
This will be much less laborious than storing it in a YAML file that
needs a deploy every time it’s updated.
An organisation can now have:
- domains which we can use to automatically associate services with it
(eg anyone whose email address ends in `dwp.gsi.gov.uk` gets services
they create associated to the DWP organisation)
- default letter branding for any new services
- default email branding for any new services
It should be nullable so we can tell whether someone has answered the
question already or not.
No real users have entered data into this column yet, so it’s fine to
wipe it.
Changed the user_to_service mapping table into a model called
ServiceUser. When looking at users who have permission for a folder
we are only interested in users for a particular service, not all users,
so we can use the ServiceUser model to access folder permissions.
Added a user_folder_permissions table which contains the service_id,
user_id and template_folder_id. There are links between
user_folder_permissions and TemplateFolder, and between
user_folder_permissions and ServiceUser.
It makes most sense to collect this at the same time as the estimated
volumes. Which means we need to store it somewhere; we can’t put it
straight into the ticket.
This will make it easier to do analysis on the data. Almost all users
are submitting data in a numerical format now anyway, because we ask the
question in a sensible way.
When a service go live we ask people for their estimated sending
volumes. At the moment we only put this in the ticket, and store it in
a spreadsheet.
This means that a service can
- say they want to go live
- say they are sending 100,000 emails per year
- not have created any email templates
- still see ‘create templates’ as ‘completed’ in the go live checklist
If we store this data against the service we can collect it earlier, and
then use it to determine automatically what kind of templates the user
needs to create before their go live checklist can be considered
complete.
Also change constraint on template postage so it cannot be null for
letters
Also add postage to all letters in template_history to respect new constraint
However, until we can create a letter without a logo, we will still default to hm-government, because the dvla_organisation is set on the service.
This does simplify the code.
Also removed the inserts to letter_branding in the data migration file, because we can deploy this before the rest of the work is finished. But we will need to do it later.
We've already performed this update on production since you need to create the index concurrently, which is not allowed from the alembic script. For that reason we are checking if the index exists.
Running an update before setting the column default value reduces
the time the table is locked (since most rows don't have a NULL
value anymore), but the migration takes slightly longer to run
overall.
a TemplateFolder has a service, a name, and a parent. Parent is a
nullable foreign key pointing to another TemplateFolder instance. We
don't do any checks here for cyclical or otherwise invalid folder
structures so keep your data clean, folks!
Unsurprisingly, a Template can be part of a TemplateFolder - there's a
mapping class (template_folder_map to avoid giving it a dumb name) -
this mapping table shouldn't be interacted with directly - rather, you
should use the `Template.folder` or `TemplateFolder.templates`
relationship.
- pass new, sanitised pdf for sending
- move invalid pdfs to a newly created bucket
- set status fro notifications that failed pdf validation to a new status validation-failed
- adjust existing tests
Added a filename column to the dvla_organisation table and populated it
with the filenames that are currently hard-coded in template-preview.
The filenames for letter logos are going to be stored in the database,
instead of in template-preview.
We want to bring the start dates for first class letter rates forward by
a month so that we don't see billing errors when sending first class letters now.
(The feature will still go live at the planned time - this is to let us test things
beforehand.)
we had an issue where the notification postage constraint command ran
into a deadlock, after trying to acquire two exclusive access locks on
large frequently modified/read tables.
To avoid this happening, we've had to split the upgrade script into
three - one script to apply the not-valid constraint to notifications
table, one for notification_history, and a third to validate the two
constraints.
Note: The first two scripts acquire exclusive access locks, but the
third only needs a row by row lock.
since this involves changing the exsiting alembic upgrades, if you've
upgraded your db you'll need to run the following three commands to
revert your database to a previous good state.
```
alter table notifications drop constraint chk_notifications_postage_null;
alter table notification_history drop constraint chk_notification_history_postage_null;
update alembic_version set version_num = '0229_new_letter_rates';
```
There are two fun quirks of postgres/sql that we need to work around:
* any `x = y` where x or y is NULL returns NULL, rather than false.
* check constraints accept NULL or true values as good.
so, the check `postage in ('first', 'second')` returns `null` rather
than `false` when postage is null itself. This surprisingly passes the
check constraint. To get around this, we have to add an explicit not
null check as well.
A not valid constraint only checks against new rows, not existing rows.
We can call VALIDATE CONSTRAINT against this new constraint to check
the old rows (which we know are good, having run the command from
74961781). Adding a normal constraint acquires an ACCESS EXCLUSIVE
lock, but validate constraint only needs a SHARE UPDATE EXCLUSIVE lock.
see 9d4b8961 and 0a50993f for more information on marking constraints
as not valid.
