Improve and clarify large task error handling

Previously we were catching one type of exception if something went
wrong adding a notification to the queue for high volume services.
In reality there are two types of exception so this adds a second
handler to cover both.

For context, this is code we changed experimentally as part of the
upgrade to Celery 5 [1]. At the time we didn't check how the new
exception compared to the old one. It turns out they behaved the
same and we were always vulnerable to the scenario now covered by
the second exception, where the behaviour has changed in Celery 5 -
testing with a large task invocation gives...

Before (Celery 3, large-ish task):

    'process_job.apply_async(["a" * 200000])'...

    boto.exception.SQSError: SQSError: 400 Bad Request
    <?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>InvalidParameterValue</Code><Message>One or more parameters are invalid. Reason: Message must be shorter than 262144 bytes.</Message><Detail/></Error><RequestId>96162552-cd96-5a14-b3a5-7f503300a662</RequestId></ErrorResponse>

Before (Celery 3, very large task):

    <hangs forever>

After (Celery 5, large-ish task):

    botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the SendMessage operation: One or more parameters are invalid. Reason: Message must be shorter than 262144 bytes.

After (Celery 5, very large task):

    botocore.parsers.ResponseParserError: Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
    b'HTTP content length exceeded 1662976 bytes.'

[1]: 29c92a9e54
This commit is contained in:
Ben Thorner
2021-11-08 10:38:53 +00:00
parent 98b6c1d67d
commit 1872854a4e
2 changed files with 14 additions and 5 deletions

View File

@@ -1068,13 +1068,21 @@ def test_post_notifications_saves_email_or_sms_to_queue(client, notify_db_sessio
assert len(Notification.query.all()) == 0
@pytest.mark.parametrize("exception", [
botocore.exceptions.ClientError({'some': 'json'}, 'some opname'),
botocore.parsers.ResponseParserError('exceeded max HTTP body length'),
])
@pytest.mark.parametrize("notification_type", ("email", "sms"))
def test_post_notifications_saves_email_or_sms_normally_if_saving_to_queue_fails(
client, notify_db_session, mocker, notification_type
client,
notify_db_session,
mocker,
notification_type,
exception
):
save_task = mocker.patch(
f"app.celery.tasks.save_api_{notification_type}.apply_async",
side_effect=botocore.exceptions.ClientError({'some': 'json'}, 'some opname')
side_effect=exception,
)
mock_send_task = mocker.patch(f'app.celery.provider_tasks.deliver_{notification_type}.apply_async')