From 86c2ec2af879d637d838b351faa0645b280039ef Mon Sep 17 00:00:00 2001 From: Alexey Bezhan Date: Tue, 18 Sep 2018 11:25:42 +0100 Subject: [PATCH] Stop gunicorn from terminating eventlet workers on timeout According to gunicorn `timeout` docs, async workers shouldn't be terminated after the request time exceeds the specified timeout as long as the worker process is still communicating. This makes sense for async workers since they generally don't block other requests from processing. This is the behaviour we've seen on notifications-api - requests can sometimes take much longer than the default 30 second timeout and still succeed. For the admin app however gunicorn has been shutting down workers after 30s with `[CRITICAL] WORKER TIMEOUT`. This results in a 502 response from the admin app. Most of these requests fail because the underlying requests to the api is taking longer than 30 seconds, however we haven't seen this in the logs originally since unlike the admin app, api requests succeed (even though their response is no longer needed). This seems like a bug in newer versions of gunicorn, downgrading it to the version that the api is currently using solves the problem by allowing admin requests to take longer than 30 seconds. (Tested by trying requests with a `time.sleep(30)` locally). It's not clear what exactly the bug in gunicorn is, but this also potentially unblocks eventlet workers and allows them to process more than one request at a time. --- requirements-app.txt | 2 +- requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements-app.txt b/requirements-app.txt index 5235d6de5..b909ad86c 100644 --- a/requirements-app.txt +++ b/requirements-app.txt @@ -13,7 +13,7 @@ pyexcel-xls==0.5.8 pyexcel-xlsx==0.5.6 pyexcel-ods3==0.5.2 pytz==2018.5 -gunicorn==19.8.1 +gunicorn==19.7.1 # >19.8 stops eventlet workers after a timeout whitenoise==4.0 #manages static assets eventlet==0.24.1 notifications-python-client==5.1.0 diff --git a/requirements.txt b/requirements.txt index 2f5706501..3eb435a5f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,7 +15,7 @@ pyexcel-xls==0.5.8 pyexcel-xlsx==0.5.6 pyexcel-ods3==0.5.2 pytz==2018.5 -gunicorn==19.8.1 +gunicorn==19.7.1 # >19.8 stops eventlet workers after a timeout whitenoise==4.0 #manages static assets eventlet==0.24.1 notifications-python-client==5.1.0