David McDonald 08c4f3f4d6 Do not delete s3 letters if 'created' or 'sending' ...

A few weeks ago, we deleted some pdf letters that had reached their
retention period. However, these letters were in the 'created' state so
it's very arguable that we should not have deleted them because we were
expecting to resend them and were unable to. Part of the reason for this
is that we marked the letters back to `created` as the status but we did
not nullify the `sent_at` timestamp, meaning the check on
ebb43082d5/app/dao/notifications_dao.py (L346)
did not catch it.

This commit does three things

Firstly, it changes the check from looking at `sent_at` to looking at
`notification_status`. Status is a much more reliable way of determinig
if the letter should be sent or not, in case we forget to nullify the
`sent_at` field in the future.

Secondly, it changes it so letters still in `sending` will not be
deleted from s3. This is important because this protects against the
case were we hand the letter to DVLA, they accept it and it gets marked
as `sending` and then the letter is deleted before they come back to us
and say there are problems and we need to resend it to them.

Thirdly, it improves test coverage for letter deleting for a range of
letter statuses.

Note, this will change will NOT stop letters in `created` or `sending`
from still being removed from the `notifications` table and put in the
`notification_history` table. It would be nice to do this too but not as
part of this commit.

It does mean there could be a case were a letter is still in sending
when it gets moved to `notification_history` and we retain the PDF. Then
the letter is updated to `delivered`. However at this point, although
the letter has reached a final state, nothing will come along and tell
s3 to delete that PDF. However, as a backup, there is a 90 day default
deletion on letters in the `production-letters-pdf` bucket so it may be
kept slightly longer than intended but not forever. This feels like a
tolerable risk, that already existed for letters in `created` state` and
is likely a better situation than deleting the PDF too early.

2020-12-15 17:29:58 +00:00

..

app

Do not delete s3 letters if 'created' or 'sending'

2020-12-15 17:29:58 +00:00

__init__.py

…

conftest.py

add broadcast_event table

2020-07-28 12:10:18 +01:00

test_all_queues_used.py

…