darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2025-12-31 21:01:39 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add more memory for the sender and letter workers

Browse Source 
On monday, we had a build of emails in the email queue that weren't
getting picked up by the sender worker and causing delays.

After further investigation with Andy from the PaaS, we believe the
following happened.

We received a bunch of traffic at 8:30ish which consisted of some
very large emails in terms of their length and complexity. The amount
of memory used by the app instances got very high and a few apps
crashed due to OOM (recorded by 5 cf app event crashes). When new
app instances tried to spin up, they weren't able to as they
potentially also ran out of memory immediately.

This left us in the position of having fewer app instances than we
needed, on top of which they were all using a very large amount of
CPU and may have been limited how quickly an individual app
instance would process tasks. This meant that we were overall
processing fewer tasks then we needed to and our queue of emails
started to build up.

So it appears our sender workers did not have the memory available that
they needed. By looking at a graph for the past 30 days of memory usage
on the sender workers, we see that it on several days breached 90%
memory usage for long periods of time. This in combination of the
hypothesis above of what happened leads us to decide that we want to
give the app instances a bigger memory quota so it has been upped from
3GB to 4GB.

Whilst doing, I also looked at long term memory usage graphs for our
other workers and saw that the letters worker was similarly close to
around 90% of memory used so have taken the opportunity to bump that
too.
This commit is contained in:
David McDonald
2020-12-24 15:03:39 +00:00
parent b8a191c3e3
commit 1ac3ca250c
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
manifest.yml.j2
View File

@@ -52,7 +52,7 @@
'notify-delivery-celery-beat': {'memory': '128M'},
'notify-delivery-worker-jobs': {},
'notify-delivery-worker-research': {},
'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '3G'},
'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '4G'},
'notify-delivery-worker-periodic': {},
'notify-delivery-worker-reporting': {
'additional_env_vars': {
@@ -61,7 +61,7 @@
}
},
'notify-delivery-worker-priority': {},
'notify-delivery-worker-letters': {},
'notify-delivery-worker-letters': {'memory': '2G'},
'notify-delivery-worker-retry-tasks': {},
'notify-delivery-worker-internal': {},
'notify-delivery-worker-receipts': {},