Stop killing reporting processes after each task

Previously we think this setting was necessary to avoid a memory
leak [1], but it's unclear if this is still an issue:

- We've advanced two major versions of Celery.
- Some of the tasks are now quicker and leaner.

Restarting worker sub-processes after each task is a big problem
for performance, as we move towards parallelising our reporting.

This is something of a test to see if we can manage without this
setting. Note that we need to unset the variable manually:

   cf unset-env notify-delivery-worker-reporting CELERYD_MAX_TASKS_PER_CHILD

In the worst case we can always re-run any failed tasks. To check
the worker is still behaving as expected, we can:

- Monitor CPU / memory graphs for it.
- Check `cf events` for unexpected restarts / crashes.
- Compare numbers of task completion logs to previous days.
- Check the number of new billing / status rows looks right.

[1]: ad419f7592
This commit is contained in:
Ben Thorner
2022-01-24 12:45:24 +00:00
parent 4a90cde701
commit 7ad0c4103a
2 changed files with 0 additions and 5 deletions

View File

@@ -344,10 +344,6 @@ class Config(object):
if os.getenv('CELERYD_PREFETCH_MULTIPLIER'):
CELERY['worker_prefetch_multiplier'] = os.getenv('CELERYD_PREFETCH_MULTIPLIER')
# on reporting worker, restart workers after each task is executed to help prevent memory leaks
if os.getenv('CELERYD_MAX_TASKS_PER_CHILD'):
CELERY['worker_max_tasks_per_child'] = int(os.getenv('CELERYD_MAX_TASKS_PER_CHILD'))
FROM_NUMBER = 'development'
STATSD_HOST = os.getenv('STATSD_HOST')