Removes notifications.template_id foreign key and replaces it with
a composite foreign key constraint to TemplateHistory using the
existing notification columns for template ID and template version.
Foreign key constraint is created as NOT VALID to avoid locking the
notifications table while postgres verifies that existing records
don't break the constraint. From postgres docs:
> If the constraint is marked NOT VALID, the potentially-lengthy initial
> check to verify that all rows in the table satisfy the constraint is
> skipped. The constraint will still be enforced against subsequent
> inserts or updates (that is, they'll fail unless there is a matching
> row in the referenced table, in the case of foreign keys; and they'll
> fail unless the new row matches the specified check constraints). But
> the database will not assume that the constraint holds for all rows
> in the table, until it is validated by using the VALIDATE CONSTRAINT
> option.
VALIDATE CONSTRAINT will be issued as a separate migration in a
follow-up PR.
Adds test for:
* checking the template version foreign key constraint
* checking that template changes don't affect existing notifications
* notification statistics aren't affected by different template versions
* notification stats always return the current template name
Relationship attribute is not used by the application code, so we
can remove it without replacing it with a TemplateHistory one.
This also updates the foreign key constraint to refer to the composite
primary key for the TemplateHistory records.
TemplateHistory objects need to be connected to the template's
TemplateRedacted record. This requires setting up a new SQLAlchemy
relationship to allow accessing redact_personalisation from
TemplateHistory instances.
We can avoid creating a foreign key in this case by setting up an
explicit `primaryjoin` expression. Since TemplateHistory.id is
created from Template.id and TemplateRedacted.template_id is
already a foreign key to Template.id the foreign key should always
be valid even without a DB constraint.
Notifications.serialize calls `Notifications.template.get_link`, so
we need TemplateHistory objects to generate their API URLs.
This generates a link to the `/v2/` version of the endpoint.
`create_custom_template` is not using `dao_create_template` since
it explicitly sets a template id. Notifications.template relationship
now refers to a TemplateHistory objects, so we need to make sure that
`create_custom_template` creates a matching TemplateHistory record
when it creates a Template.
`Notification.template` changed from being a Template relationship
to TemplateHistory. When a relationship attribute is being assigned,
SQLAlchemy checks that the assigned value type matches the relationship
type. Since most tests at the moment create a notification using a
Template instance this check fails.
Rewriting the tests to use TemplateHistory objects would require
changes to the majority of tests. Instead, when creating a notification
objects we assign the foreign key attributes directly. This skips the
SQLAlchemy type check, but we still get the constraint check on the
foreign keys, so a matching TemplateHistory object needs to exist in
the database.
`Notification.template_history` relationship has been removed but
we want to keep the `template_history` key in existing notification
serializations, so we serialize it from `Notifications.template`.
This keeps the data format the same, but both `template` and
`template_history` keys will now contain data from the `TemplateHistory`
instance.
Each notification instance is connected to a specific version of the
template that was used to generate notification content. Template
versions are stored in TemplateHistory, so that's the table we need
to use for Notifications.template relationship.
Currently, using Notifications.template returns the latest template
version, which leads to incorrect content being returned by the API
if the template has been updated.
This change removes Notifications.template_history attribute and
replaces template relationship to refer to TemplateHistory instance
based on the notification's template ID and version.
There are some Null template_ids in the production database which was
causing a failure as the stats_template_usage_by_month has a constraint
that it should not be null. This update only adds populated template_ids
- Updated Celery task
- Add a new test to test for null template_ids and not try to add them
to the stats
- Modified the services_dao to return an int instead of a datetime to
make usage easier and removed the BST function on year as it is not
relevant for year
- Improved tests do there is less logic by ordering the result so there
is less reliance on the template id
- Renamed variable in stats_template_usage_by_month_dao.py to make it
consistent with the method
- Some tests for failing because of an import which was overriding the
one above it in test_scheduled_tasks.py
- Import error fixed in test_services_dao.py after a rename of a method
Currently some pages are timing out due to the time it takes to perform
database queries. This is an attempt to improve the performance by
performing the query against the notification history table once a day
and use the notification table for a delta between midnight and the when
the page is run and combine the results.
- Added Celery task for doing the work
- Added a dao to handle the insert and update of the stats table
- Updated tests to test the new functionality
