Commit Graph

5 Commits

Author SHA1 Message Date
Alexey Bezhan
c2e15d4ee2 Allow retry exception to propagate from ses callback task
Celery `self.retry` raises an exception to communicate that the task
needs to be retried. Since our ses task is wrapped in a catch-all
except block it logs that exception as an error before retrying.

Handling Retry class separately allows us to raise it without logging
the traceback.
2019-02-25 13:25:50 +00:00
Alexey Bezhan
2932b44eb8 Add retries for SES callbacks for recent notifications
We've seen errors caused by what we suspect is a race condition when
SES callback processing tries to look up the notification before the
sender worker has saved notification reference from the SES POST
response to the database.

This adds a retry for SES callback task if the notification was not
found and the message is less than 10 minutes old and removes the
error log message for notifications older than 3 days (since they
might no longer exist in the notifications table and would've been
marked as failure by then either way).

In order to be able to call retry and silence the error log based on
notification time this change inlines `process_ses_response` and
`update_notification_by_reference` functions into the celery task.
It also removes a lot of defensive error-handling that doesn't appear
to have been triggered in the last few months (for things like missing
keys in SES callback data).
2019-02-25 10:36:37 +00:00
Leo Hemsted
267c4fc07b bump requirements, fix pyflake8 things, unpin botocore/awscli 2018-11-07 13:39:08 +00:00
Richard Chapman
d855b4e4ec Removed statsd from the api and use the statsd in the utils library.
The statsd code was added to the utils library a while ago, uses the
statsd from the util library and therefore consolidates the code into
once place.
2018-02-06 09:52:15 +00:00
venusbb
489f43a2c9 rename callback_tasks.py to process_ses_receipts.py
create service_callback_tasks.py for tasks to send delivery statuses to services
2017-12-01 16:15:21 +00:00