Reduce pool size from 30 to 15 connections

Having a pool size of 30 connections means that if we receive a big
number of requests, with the current configuration, the API would end up
holding onto 30 connections per worker * 4 workers per instance * 35
instances = 4200 connections. With a limit of 5000 connections, this
means that we would only have 800 connections to share between the
workers or for overflow usage (btw, even the overflow for the API would
take us above the 5000 limit - 10 overflow connections per worker * 4 *
35 = 1400 connections, total 5600 _only_ for the API).

During our load tests this led to a deadlock situation where nothing
could retrieve connections to deal with a queue build-up.

The reduced pool size allowed for a much more graceful degradation of
the service where, after significant load we would increase the response
times but still manage to serve all the requests.
This commit is contained in:
sakisv
2021-12-23 19:28:17 +02:00
parent 77084533fb
commit ad8cf3f3a6

View File

@@ -2,7 +2,7 @@
'notify-api': {
'NOTIFY_APP_NAME': 'api',
'disk_quota': '2G',
'sqlalchemy_pool_size': 30,
'sqlalchemy_pool_size': 15,
'additional_env_vars': {
'STATSD_HOST': None
},