Scale timeout task to work on arbitrary volumes

Previously this was limited to 500K notifications. While we don't
expect to reach this limit, it's not impossible e.g. if we had a
repeat of the incident where one of our providers stopped sending
us status updates. Although that's not great, it's worse if our
code can't cope with the unexpectedly high volume.

This reuses the technique we have elsewhere [1] to keep processing
in batches until there's nothing left. Specifying a cutoff point
means the total amount of work to do can't keep growing.

[1]: 2fb432adaf/app/dao/notifications_dao.py (L441)

app/celery/nightly_tasks.py

@@ -113,15 +113,13 @@ def delete_letter_notifications_older_than_retention():
 @notify_celery.task(name='timeout-sending-notifications')
 @cronitor('timeout-sending-notifications')
 def timeout_notifications():
-    # TEMPORARY: re-run the following code over small batches of notifications
-    # so that we can cope with a high volume that need processing. We've changed
-    # dao_timeout_notifications to return up to 100K notifications, so this task
-    # will operate on up to 500K - normally we only get around 20K.
+    notifications = ['dummy value so len() > 0']
+
     cutoff_time = datetime.utcnow() - timedelta(
         seconds=current_app.config.get('SENDING_NOTIFICATIONS_TIMEOUT_PERIOD')
     )
 
-    for _ in range(0, 5):
+    while len(notifications) > 0:
         notifications = dao_timeout_notifications(cutoff_time)
 
         for notification in notifications:
@@ -130,11 +128,6 @@ def timeout_notifications():
         current_app.logger.info(
             "Timeout period reached for {} notifications, status has been updated.".format(len(notifications)))
 
-        if len(notifications) < 100000:
-            return
-
-    raise RuntimeError("Some notifications may still be in sending.")
-
 
 @notify_celery.task(name="delete-inbound-sms")
 @cronitor("delete-inbound-sms")