retry sending broadcasts

Retry tasks if they fail to send a broadcast event. Note that each task
tries the regular proxy and the failover proxy for that provider. This
runs a bit differently than our other retries:

Retry with exponential backoff. Our other tasks retry with a fixed delay
of 5 minutes between tries. If we can't send a broadcast, we want to try
immediately. So instead, implement an exponential backoff (1, 2, 4, 8,
... seconds delay). We can't delay for longer than 310 seconds due to
visibility timeout settings in SQS, so cap the delay at that amount.

Normally we give up retrying after a set amount of retries (often 4
hours). As broadcast content is much more important than normal
notifications, we don't ever want to give up on sending them to phones...

...UNLESS WE DO!

Sometimes we do want to give up sending a broadcast though! Broadcasts
have an expiry time, when they stop showing up on peoples devices, so if
that has passed then we don't need to send the broadcast out.

Broadcast events can also be superceded by updates or cancels. Check
that the event is the most recent event for that broadcast message, if
not, give up, as we don't want to accidentally send out two conflicting
events for the same message.

app/clients/cbc_proxy.py

@@ -25,7 +25,11 @@ from app.utils import DATETIME_FORMAT, format_sequential_number
 #    the preceeding Alert message in the previous_provider_messages field
 
 
-class CBCProxyException(Exception):
+class CBCProxyFatalException(Exception):
+    pass
+
+
+class CBCProxyRetryableException(Exception):
     pass
 
 
@@ -115,7 +119,9 @@ class CBCProxyClientBase(ABC):
         if not result:
             failover_result = self._invoke_lambda(self.failover_lambda_name, payload)
             if not failover_result:
-                raise CBCProxyException(f'Lambda failed for both {self.lambda_name} and {self.failover_lambda_name}')
+                raise CBCProxyRetryableException(
+                    f'Lambda failed for both {self.lambda_name} and {self.failover_lambda_name}'
+                )
 
         return result