From e7efeec30974d9d5d43c65f91539039b2d196978 Mon Sep 17 00:00:00 2001 From: Rebecca Law Date: Wed, 1 Dec 2021 11:40:18 +0000 Subject: [PATCH] Increase the concurrency for the delivery-worker-reporting TL;DR After a chat with some team members we've decided to double the concurrency of the delivery-worker-reporting app to 4 from 2. Looking at the memory usage during the reporting task runs we don't believe this to be a risk. There are some other things to look at, but this could be a quick win in the short term. Longer read: Every night we have 2 "reporting" tasks that run. - create-nightly-billing starts at 00:15 - populates data for ft_billing for the previous days. - 4 days for email - 4 days for sms - 10 days for letters - create-nightly-notification-status starts at 00:30 - populates data for ft_notification - 4 days for email - 4 days for sms - 10 days for letters These tasks are picked up by the `notify-delivery-worker-reporting` app, we run 3 instances with a concurrency = 2. This means that we have 6 worker threads that pick up the 18 tasks created at 00:15 and 00:30. Each celery main thread picks up 10 tasks of the queue, the 2 worker threads start working on a task and acknowledge the task to SQS. Meanwhile the other 8 tasks wait in the internal celery queue and are no acknowledgement is sent to SQS. As each task is complete a worker picks up a new thread, acknowledges the task. If a task is kept in the Celery internal queue for longer than 5 minutes the visibility timeout in SQS will assume the task has not completed and put the task back on the availability queue, therefore creating a duplicate task. At some point all the tasks are completed, some are completed twice. --- scripts/paas_app_wrapper.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/paas_app_wrapper.sh b/scripts/paas_app_wrapper.sh index cd8e8c1e9..7aeb46868 100755 --- a/scripts/paas_app_wrapper.sh +++ b/scripts/paas_app_wrapper.sh @@ -29,7 +29,7 @@ case $NOTIFY_APP_NAME in -Q periodic-tasks 2> /dev/null ;; delivery-worker-reporting) - exec scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=2 \ + exec scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=4 \ -Q reporting-tasks 2> /dev/null ;; delivery-worker-priority)