darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2026-09-08 07:18:26 -04:00
Code Issues Packages Projects Releases Wiki Activity

bulk_update_mappings adr

Browse Source
This commit is contained in:
Kenneth Kehl
2025-01-22 09:40:59 -08:00
parent e6899ef9d0
commit 669f4e9bfd
2 changed files with 37 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/adrs/0010-adr-celery-pool-support-best-practice.md
View File

@@ -1,7 +1,7 @@
# Make best use of celery worker pools
Status: N/A
Date: N/A
Status: Accepted
Date: 7 January 2025
### Context
Our API application started with initial celery pool support of 'prefork' (the default) and concurrency of 4. We continuously encountered instability, which we initially attributed to a resource leak. As a result of this we added the configuration `worker-max-tasks-per-child=500` which is a best practice. When we ran a load test of 25000 simulated messages, however, we continued to see stability issues, amounting to a crash of the app after 4 hours requiring a restage. Based on running `cf app notify-api-production` and observing that `cpu entitlement` was off the charts at 10000% to 12000% for the works, and after doing some further reading, we came to the conclusion that perhaps `prefork` pool support is not the best type of pool support for the API application.
@@ -10,8 +10,12 @@ The problem with `prefork` is that each process has a tendency to hang onto the
### Decision
We decided to try to the 'threads' pool support with increased concurrency.
### Consequences
We saw an immediate decrease in CPU usage of about 70% with no adverse consequences.
### Author
@kenkehl