From 1ac3ca250cbec00eb6de4ba8304b1c1b90fec210 Mon Sep 17 00:00:00 2001 From: David McDonald Date: Thu, 24 Dec 2020 15:03:39 +0000 Subject: [PATCH] Add more memory for the sender and letter workers On monday, we had a build of emails in the email queue that weren't getting picked up by the sender worker and causing delays. After further investigation with Andy from the PaaS, we believe the following happened. We received a bunch of traffic at 8:30ish which consisted of some very large emails in terms of their length and complexity. The amount of memory used by the app instances got very high and a few apps crashed due to OOM (recorded by 5 cf app event crashes). When new app instances tried to spin up, they weren't able to as they potentially also ran out of memory immediately. This left us in the position of having fewer app instances than we needed, on top of which they were all using a very large amount of CPU and may have been limited how quickly an individual app instance would process tasks. This meant that we were overall processing fewer tasks then we needed to and our queue of emails started to build up. So it appears our sender workers did not have the memory available that they needed. By looking at a graph for the past 30 days of memory usage on the sender workers, we see that it on several days breached 90% memory usage for long periods of time. This in combination of the hypothesis above of what happened leads us to decide that we want to give the app instances a bigger memory quota so it has been upped from 3GB to 4GB. Whilst doing, I also looked at long term memory usage graphs for our other workers and saw that the letters worker was similarly close to around 90% of memory used so have taken the opportunity to bump that too. --- manifest.yml.j2 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/manifest.yml.j2 b/manifest.yml.j2 index 8085644cd..eb8aab229 100644 --- a/manifest.yml.j2 +++ b/manifest.yml.j2 @@ -52,7 +52,7 @@ 'notify-delivery-celery-beat': {'memory': '128M'}, 'notify-delivery-worker-jobs': {}, 'notify-delivery-worker-research': {}, - 'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '3G'}, + 'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '4G'}, 'notify-delivery-worker-periodic': {}, 'notify-delivery-worker-reporting': { 'additional_env_vars': { @@ -61,7 +61,7 @@ } }, 'notify-delivery-worker-priority': {}, - 'notify-delivery-worker-letters': {}, + 'notify-delivery-worker-letters': {'memory': '2G'}, 'notify-delivery-worker-retry-tasks': {}, 'notify-delivery-worker-internal': {}, 'notify-delivery-worker-receipts': {},