darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2025-12-15 09:42:38 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
e3e067c79544b185243a37a9faa9c645ee5dcbb5
notifications-api/app/celery/process_ses_receipts_tasks.py

85 lines
3.3 KiB
Python
Raw Normal View History

Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from datetime import datetime, timedelta
import iso8601
Allow retry exception to propagate from ses callback task Celery `self.retry` raises an exception to communicate that the task needs to be retried. Since our ses task is wrapped in a catch-all except block it logs that exception as an error before retrying. Handling Retry class separately allows us to raise it without logging the traceback.
2019-02-25 13:25:50 +00:00
from celery.exceptions import Retry
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from flask import current_app, json
from sqlalchemy.orm.exc import NoResultFound
update research mode email callbacks to add process-ses-response task to queue this involved: * moving that task to callback_tasks to prevent circular imports * updating the dummy research mode callbacks (with actual SNS messages from the ses simulator emails) * refactoring tests
2017-11-17 13:41:45 +00:00
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app import notify_celery, statsd_client
from app.clients.email.aws_ses import get_aws_responses
Run auto-correct on app/ and tests/
2021-03-10 13:55:06 +00:00
from app.config import QueueNames
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app.dao import notifications_dao
Run auto-correct on app/ and tests/
2021-03-10 13:55:06 +00:00
from app.models import NOTIFICATION_PENDING, NOTIFICATION_SENDING
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
from app.notifications.notifications_ses_callback import (
Run auto-correct on app/ and tests/
2021-03-10 13:55:06 +00:00
_check_and_queue_callback_task,
_check_and_queue_complaint_callback_task,
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
determine_notification_bounce_type,
handle_complaint,
)
update research mode email callbacks to add process-ses-response task to queue this involved: * moving that task to callback_tasks to prevent circular imports * updating the dummy research mode callbacks (with actual SNS messages from the ses simulator emails) * refactoring tests
2017-11-17 13:41:45 +00:00
@notify_celery.task(bind=True, name="process-ses-result", max_retries=5, default_retry_delay=300)
def process_ses_results(self, response):
try:
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
ses_message = json.loads(response['Message'])
notification_type = ses_message['notificationType']
Add notification ID to SES bounce reason At the moment we log everytime we get a bounce from SES, however we don't link it to a particular notification so it's hard to know for what sub reason a notifcation did not deliver by looking at the logs. This commit changes this by now looking the bounce reason after we have found the notification ID and including them together. So if you know search for a notification ID in Kibana, you will see full logs for why it failed to deliver.
2020-11-20 14:10:13 +00:00
bounce_message = None
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
if notification_type == 'Bounce':
Add notification ID to SES bounce reason At the moment we log everytime we get a bounce from SES, however we don't link it to a particular notification so it's hard to know for what sub reason a notifcation did not deliver by looking at the logs. This commit changes this by now looking the bounce reason after we have found the notification ID and including them together. So if you know search for a notification ID in Kibana, you will see full logs for why it failed to deliver.
2020-11-20 14:10:13 +00:00
notification_type, bounce_message = determine_notification_bounce_type(notification_type, ses_message)
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
elif notification_type == 'Complaint':
_check_and_queue_complaint_callback_task(*handle_complaint(ses_message))
return True
aws_response_dict = get_aws_responses(notification_type)
notification_status = aws_response_dict['notification_status']
reference = ses_message['mail']['messageId']
try:
Rename function for clarity
2020-03-27 15:48:54 +00:00
notification = notifications_dao.dao_get_notification_or_history_by_reference(reference=reference)
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
except NoResultFound:
message_time = iso8601.parse_date(ses_message['mail']['timestamp']).replace(tzinfo=None)
Downgrade log level for missing notifications in SES receipt The timestamps available in the SES receipt don't always correspond to the time the notification has been sent. We've seen callbacks with a current timestamp in both 'mail' and 'bounce' objects that referenced a notification sent a week ago, which means we can't rely on it to skip archived notifications. One possible approach would be to look up the notification reference in the notification_history table, but this goes against our plans to stop relying on it in the future. This changes the SES receipts logic to retry missing notifications once (if the callback timestamp is within the last 5 minutes the task will retry after a 5 minute delay) to capture callbacks arriving before the notification reference has been persisted to the DB. Otherwise, we log the missing notification as a warning instead of error.
2019-03-06 11:35:32 +00:00
if datetime.utcnow() - message_time < timedelta(minutes=5):
Add better logging to understand callback delivery cases In particular, we want to know how often callbacks arrive before the notification being persisted
2020-06-04 15:43:03 +01:00
current_app.logger.info(
f"notification not found for reference: {reference} (update to {notification_status}). "
f"Callback may have arrived before notification was persisted to the DB. Adding task to retry queue"
)
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
self.retry(queue=QueueNames.RETRY)
Downgrade log level for missing notifications in SES receipt The timestamps available in the SES receipt don't always correspond to the time the notification has been sent. We've seen callbacks with a current timestamp in both 'mail' and 'bounce' objects that referenced a notification sent a week ago, which means we can't rely on it to skip archived notifications. One possible approach would be to look up the notification reference in the notification_history table, but this goes against our plans to stop relying on it in the future. This changes the SES receipts logic to retry missing notifications once (if the callback timestamp is within the last 5 minutes the task will retry after a 5 minute delay) to capture callbacks arriving before the notification reference has been persisted to the DB. Otherwise, we log the missing notification as a warning instead of error.
2019-03-06 11:35:32 +00:00
else:
current_app.logger.warning(
Add better logging to understand callback delivery cases In particular, we want to know how often callbacks arrive before the notification being persisted
2020-06-04 15:43:03 +01:00
f"notification not found for reference: {reference} (update to {notification_status})"
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
)
return
Add notification ID to SES bounce reason At the moment we log everytime we get a bounce from SES, however we don't link it to a particular notification so it's hard to know for what sub reason a notifcation did not deliver by looking at the logs. This commit changes this by now looking the bounce reason after we have found the notification ID and including them together. So if you know search for a notification ID in Kibana, you will see full logs for why it failed to deliver.
2020-11-20 14:10:13 +00:00
if bounce_message:
current_app.logger.info(f"SES bounce for notification ID {notification.id}: {bounce_message}")
Refactor process_ses_receipt It is possible a service has data rention that is smaller than the time it takes to get a delivery receipt. This PR refactors process_ses_receipt to update NotificationHistory if the Notifcation has already been purged.
2020-03-27 14:12:39 +00:00
if notification.status not in [NOTIFICATION_SENDING, NOTIFICATION_PENDING]:
notifications_dao._duplicate_update_warning(
notification=notification,
status=notification_status
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
)
Refactor process_ses_receipt It is possible a service has data rention that is smaller than the time it takes to get a delivery receipt. This PR refactors process_ses_receipt to update NotificationHistory if the Notifcation has already been purged.
2020-03-27 14:12:39 +00:00
return
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
else:
Refactor process_ses_receipt It is possible a service has data rention that is smaller than the time it takes to get a delivery receipt. This PR refactors process_ses_receipt to update NotificationHistory if the Notifcation has already been purged.
2020-03-27 14:12:39 +00:00
notifications_dao.dao_update_notifications_by_reference(
references=[reference],
update_dict={'status': notification_status}
)
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
statsd_client.incr('callback.ses.{}'.format(notification_status))
if notification.sent_at:
statsd_client.timing_with_dates('callback.ses.elapsed-time', datetime.utcnow(), notification.sent_at)
_check_and_queue_callback_task(notification)
return True
Allow retry exception to propagate from ses callback task Celery `self.retry` raises an exception to communicate that the task needs to be retried. Since our ses task is wrapped in a catch-all except block it logs that exception as an error before retrying. Handling Retry class separately allows us to raise it without logging the traceback.
2019-02-25 13:25:50 +00:00
except Retry:
raise
Add retries for SES callbacks for recent notifications We've seen errors caused by what we suspect is a race condition when SES callback processing tries to look up the notification before the sender worker has saved notification reference from the SES POST response to the database. This adds a retry for SES callback task if the notification was not found and the message is less than 10 minutes old and removes the error log message for notifications older than 3 days (since they might no longer exist in the notifications table and would've been marked as failure by then either way). In order to be able to call retry and silence the error log based on notification time this change inlines `process_ses_response` and `update_notification_by_reference` functions into the celery task. It also removes a lot of defensive error-handling that doesn't appear to have been triggered in the last few months (for things like missing keys in SES callback data).
2019-02-13 11:35:31 +00:00
except Exception as e:
current_app.logger.exception('Error processing SES results: {}'.format(type(e)))
celery.task.retry exc param should be a throwable. This causes an issue when it hits the max retry limit, and tries to throw your exception to let you deal with it - at this point it was moaning because we pass in a string if it's not defined, and we're inside an exception block celery uses that instead.
2017-11-23 13:49:52 +00:00
self.retry(queue=QueueNames.RETRY)
Reference in New Issue Copy Permalink