mirror of
https://github.com/GSA/notifications-api.git
synced 2026-02-01 07:35:34 -05:00
Stop killing reporting processes after each task
Previously we think this setting was necessary to avoid a memory
leak [1], but it's unclear if this is still an issue:
- We've advanced two major versions of Celery.
- Some of the tasks are now quicker and leaner.
Restarting worker sub-processes after each task is a big problem
for performance, as we move towards parallelising our reporting.
This is something of a test to see if we can manage without this
setting. Note that we need to unset the variable manually:
cf unset-env notify-delivery-worker-reporting CELERYD_MAX_TASKS_PER_CHILD
In the worst case we can always re-run any failed tasks. To check
the worker is still behaving as expected, we can:
- Monitor CPU / memory graphs for it.
- Check `cf events` for unexpected restarts / crashes.
- Compare numbers of task completion logs to previous days.
- Check the number of new billing / status rows looks right.
[1]: ad419f7592
This commit is contained in:
@@ -344,10 +344,6 @@ class Config(object):
|
||||
if os.getenv('CELERYD_PREFETCH_MULTIPLIER'):
|
||||
CELERY['worker_prefetch_multiplier'] = os.getenv('CELERYD_PREFETCH_MULTIPLIER')
|
||||
|
||||
# on reporting worker, restart workers after each task is executed to help prevent memory leaks
|
||||
if os.getenv('CELERYD_MAX_TASKS_PER_CHILD'):
|
||||
CELERY['worker_max_tasks_per_child'] = int(os.getenv('CELERYD_MAX_TASKS_PER_CHILD'))
|
||||
|
||||
FROM_NUMBER = 'development'
|
||||
|
||||
STATSD_HOST = os.getenv('STATSD_HOST')
|
||||
|
||||
Reference in New Issue
Block a user