Rebecca Law e7efeec309 Increase the concurrency for the delivery-worker-reporting ...

TL;DR
After a chat with some team members we've decided to double the concurrency of the delivery-worker-reporting app to 4 from 2. Looking at the memory usage during the reporting task runs we don't believe this to be a risk. There are some other things to look at, but this could be a quick win in the short term.

Longer read:
Every night we have 2 "reporting" tasks that run.
- create-nightly-billing starts at 00:15
  - populates data for ft_billing for the previous days.
  - 4 days for email
  - 4 days for sms
  - 10 days for letters
- create-nightly-notification-status starts at 00:30
  - populates data for ft_notification
  - 4 days for email
  - 4 days for sms
  - 10 days for letters

These tasks are picked up by the `notify-delivery-worker-reporting` app, we run 3 instances with a concurrency = 2.
This means that we have 6 worker threads that pick up the 18 tasks created at 00:15 and 00:30.
Each celery main thread picks up 10 tasks of the queue, the 2 worker threads start working on a task and acknowledge the task to SQS. Meanwhile the other 8 tasks wait in the internal celery queue and are no acknowledgement is sent to SQS. As each task is complete a worker picks up a new thread, acknowledges the task.
If a task is kept in the Celery internal queue for longer than 5 minutes the visibility timeout in SQS will assume the task has not completed and put the task back on the availability queue, therefore creating a duplicate task.
At some point all the tasks are completed, some are completed twice.

2021-12-01 11:40:18 +00:00

2.9 KiB

Executable File

Raw Blame History

View Raw