From 5d88a1dbf46585070593d209d68e162a43e3a5a0 Mon Sep 17 00:00:00 2001 From: David McDonald Date: Mon, 27 Apr 2020 15:12:44 +0100 Subject: [PATCH] Add -Ofair setting to reporting celery app What this setting does is best described in https://medium.com/@taylorhughes/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb#d7ec This should be useful for the reporting app because tasks run by this app are long running (many seconds). Ideally this code change will mean that we are quicker to process the overnight reporting tasks, so they all finish earlier in the morning (although are not individually quicker). This is only being set on the reporting celery app because this change trying to do the minimum possible to improve the reliability and speed of our overnight reporting tasks. It may very well be useful to set this flag on all our apps, but this should be done with some more consideration as some of them will deal with much faster tasks (sub 0.5s) and so it may be still be appropriate or may not. Proper investigation would be needed. Note, the celery docs on this are also worth a read: https://docs.celeryproject.org/en/3.1/userguide/optimizing.html#optimizing-prefetch-limit. However, the language can confuse this with setting with the prefetch limit. The distinction is that prefetch grabs items off the queue, whereas the -Ofair behaviour is to do with when items have already been prefetched and then whether the master celery process straight away gives them to the child (worker) processes or not. Note, this behaviour is default for celery version 4 and above but we are still on version 3.1.26 so we have to enable it ourselves. --- scripts/paas_app_wrapper.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/paas_app_wrapper.sh b/scripts/paas_app_wrapper.sh index d574694f9..b855d9192 100755 --- a/scripts/paas_app_wrapper.sh +++ b/scripts/paas_app_wrapper.sh @@ -29,7 +29,7 @@ case $NOTIFY_APP_NAME in -Q periodic-tasks 2> /dev/null ;; delivery-worker-reporting) - exec scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 \ + exec scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Ofair \ -Q reporting-tasks 2> /dev/null ;; delivery-worker-priority)