darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2025-12-17 02:32:32 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
8da73510c2c14fc980e25e57ce347c581789c5c5
notifications-api/app/celery/process_ses_receipts_tasks.py

85 lines
3.2 KiB
Python
Raw Normal View History

Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from datetime import datetime, timedelta
import iso8601
Allow retry exception to propagate from ses callback task Celery `self.retry` raises an exception to communicate that the task needs to be retried. Since our ses task is wrapped in a catch-all except block it logs that exception as an error before retrying. Handling Retry class separately allows us to raise it without logging the traceback.
2019-02-25 13:25:50 +00:00
from celery.exceptions import Retry
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from flask import current_app, json
Removed statsd from the api and use the statsd in the utils library. The statsd code was added to the utils library a while ago, uses the statsd from the util library and therefore consolidates the code into once place.
2018-02-06 09:35:33 +00:00
from notifications_utils.statsd_decorators import statsd
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from sqlalchemy.orm.exc import NoResultFound
update research mode email callbacks to add process-ses-response task to queue this involved: * moving that task to callback_tasks to prevent circular imports * updating the dummy research mode callbacks (with actual SNS messages from the ses simulator emails) * refactoring tests
2017-11-17 13:41:45 +00:00
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app import notify_celery, statsd_client
update research mode email callbacks to add process-ses-response task to queue this involved: * moving that task to callback_tasks to prevent circular imports * updating the dummy research mode callbacks (with actual SNS messages from the ses simulator emails) * refactoring tests
2017-11-17 13:41:45 +00:00
from app.config import QueueNames
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app.clients.email.aws_ses import get_aws_responses
from app.dao import notifications_dao
from app.models import NOTIFICATION_SENDING, NOTIFICATION_PENDING
Removed statsd from the api and use the statsd in the utils library. The statsd code was added to the utils library a while ago, uses the statsd from the util library and therefore consolidates the code into once place.
2018-02-06 09:35:33 +00:00
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app.notifications.notifications_ses_callback import (
determine_notification_bounce_type,
handle_complaint,
_check_and_queue_complaint_callback_task,
_check_and_queue_callback_task,
)
update research mode email callbacks to add process-ses-response task to queue this involved: * moving that task to callback_tasks to prevent circular imports * updating the dummy research mode callbacks (with actual SNS messages from the ses simulator emails) * refactoring tests
2017-11-17 13:41:45 +00:00
@notify_celery.task(bind=True, name="process-ses-result", max_retries=5, default_retry_delay=300)
@statsd(namespace="tasks")
def process_ses_results(self, response):
try:
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
ses_message = json.loads(response['Message'])
notification_type = ses_message['notificationType']
if notification_type == 'Bounce':
notification_type = determine_notification_bounce_type(notification_type, ses_message)
elif notification_type == 'Complaint':
_check_and_queue_complaint_callback_task(*handle_complaint(ses_message))
return True
aws_response_dict = get_aws_responses(notification_type)
notification_status = aws_response_dict['notification_status']
reference = ses_message['mail']['messageId']
try:
notification = notifications_dao.dao_get_notification_by_reference(reference)
except NoResultFound:
message_time = iso8601.parse_date(ses_message['mail']['timestamp']).replace(tzinfo=None)
Downgrade log level for missing notifications in SES receipt The timestamps available in the SES receipt don't always correspond to the time the notification has been sent. We've seen callbacks with a current timestamp in both 'mail' and 'bounce' objects that referenced a notification sent a week ago, which means we can't rely on it to skip archived notifications. One possible approach would be to look up the notification reference in the notification_history table, but this goes against our plans to stop relying on it in the future. This changes the SES receipts logic to retry missing notifications once (if the callback timestamp is within the last 5 minutes the task will retry after a 5 minute delay) to capture callbacks arriving before the notification reference has been persisted to the DB. Otherwise, we log the missing notification as a warning instead of error.
2019-03-06 11:35:32 +00:00
if datetime.utcnow() - message_time < timedelta(minutes=5):
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
self.retry(queue=QueueNames.RETRY)
Downgrade log level for missing notifications in SES receipt The timestamps available in the SES receipt don't always correspond to the time the notification has been sent. We've seen callbacks with a current timestamp in both 'mail' and 'bounce' objects that referenced a notification sent a week ago, which means we can't rely on it to skip archived notifications. One possible approach would be to look up the notification reference in the notification_history table, but this goes against our plans to stop relying on it in the future. This changes the SES receipts logic to retry missing notifications once (if the callback timestamp is within the last 5 minutes the task will retry after a 5 minute delay) to capture callbacks arriving before the notification reference has been persisted to the DB. Otherwise, we log the missing notification as a warning instead of error.
2019-03-06 11:35:32 +00:00
else:
current_app.logger.warning(
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
"notification not found for reference: {} (update to {})".format(reference, notification_status)
)
return
if notification.status not in {NOTIFICATION_SENDING, NOTIFICATION_PENDING}:
notifications_dao._duplicate_update_warning(notification, notification_status)
return
notifications_dao._update_notification_status(notification=notification, status=notification_status)
if not aws_response_dict['success']:
current_app.logger.info(
"SES delivery failed: notification id {} and reference {} has error found. Status {}".format(
notification.id, reference, aws_response_dict['message']
)
)
else:
current_app.logger.info('SES callback return status of {} for notification: {}'.format(
notification_status, notification.id
))
statsd_client.incr('callback.ses.{}'.format(notification_status))
if notification.sent_at:
statsd_client.timing_with_dates('callback.ses.elapsed-time', datetime.utcnow(), notification.sent_at)
_check_and_queue_callback_task(notification)
return True
Allow retry exception to propagate from ses callback task Celery `self.retry` raises an exception to communicate that the task needs to be retried. Since our ses task is wrapped in a catch-all except block it logs that exception as an error before retrying. Handling Retry class separately allows us to raise it without logging the traceback.
2019-02-25 13:25:50 +00:00
except Retry:
raise
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
except Exception as e:
current_app.logger.exception('Error processing SES results: {}'.format(type(e)))
celery.task.retry exc param should be a throwable. This causes an issue when it hits the max retry limit, and tries to throw your exception to let you deal with it - at this point it was moaning because we pass in a string if it's not defined, and we're inside an exception block celery uses that instead.
2017-11-23 13:49:52 +00:00
self.retry(queue=QueueNames.RETRY)
Reference in New Issue Copy Permalink