If there is an invalid letter that has not been updated to validation-failed because the update-validation-failed-for-templated-letter has not been picked up off the letter-tasks queue and the collate-letter-pdfs-to-be-sent has started.

1. The number of letters that we send to DVLA will be not be correct (see 20ead82463/app/celery/letters_pdf_tasks.py (L136))
This may raise an alert with DVLA when they find we have sent them fewer letter than we have reported.
2. When we get the PDF from S3 we will get a file not found 20ead82463/app/celery/letters_pdf_tasks.py (L244)
The error will not prevent the collate task from completing but we will see an alert email for the exception and raise questions.

Although this situation is very unlikely because we have a 15 minute window between the last letter deadline date and the time we kick off the collate task we should still mitigate these issues. I updated the queries to only return letters with billable_units > 0, all valid letters should have at least 1 billable unit.

tests/app/v2/notifications/test_post_letter_notifications.py

@@ -296,7 +296,7 @@ def test_post_letter_notification_with_test_key_creates_pdf_and_sets_status_to_d
     'development',
     'preview',
 ])
-def test_post_letter_notification_with_test_key_creates_pdf(
+def test_post_letter_notification_with_test_key_creates_pdf_and_sets_status_to_sending_and_sends_fake_response_file(
         notify_api, client, sample_letter_template, mocker, env):
 
     data = {
@@ -312,7 +312,8 @@ def test_post_letter_notification_with_test_key_creates_pdf(
     }
 
     fake_create_letter_task = mocker.patch('app.celery.letters_pdf_tasks.get_pdf_for_templated_letter.apply_async')
-
+    fake_create_dvla_response_task = mocker.patch(
+        'app.celery.research_mode_tasks.create_fake_letter_response_file.apply_async')
     with set_config_values(notify_api, {
         'NOTIFY_ENVIRONMENT': env
     }):
@@ -321,6 +322,7 @@ def test_post_letter_notification_with_test_key_creates_pdf(
     notification = Notification.query.one()
 
     fake_create_letter_task.assert_called_once_with([str(notification.id)], queue='research-mode-tasks')
+    assert fake_create_dvla_response_task.called
     assert notification.status == NOTIFICATION_SENDING