The workers are working sequentially so they don't need more than one
connection per thread.
This has been tested with the sender worker and it works fine even
under significant load, so this commit will make it the default for the
other workers as well.
This was overly optimistic that 2G would be enough to handle 4 worker
processes as they are already exhausting the 2G limit.
Depending on performance we may need to tweak the memory instead/too.
We have seen problems with the service callback workers due to the db
connection pool being exhausted. When the worker picks up the task, it
makes a db query to get the notification, a query to get the callback
url, and then closes the session before it makes the 3rd party request.
However, even closing the session before the (potentially lengthy)
web request wasn't enough - we've seen significant amounts of
`sqlalchemy.exc.TimeoutError`s.
This reverts commit 2dfbd93c7e
Service callbacks are I/O bound and can take a long time if the
callback URL takes a long time to respond. This is a perfect use
case for an eventlet worker pool since it allows spawning multiple
green threads (1000 in the proposed configuration) to execute HTTP
requests concurrently without a significant increase in CPU load
or memory usage needed for adding additional worker processes.
sent by the notify service, including 2FA and email auth messages.
These should never be delayed by other messages in the queue.
Created a separate delivery app to consume the notify-internal-tasks
queue and only that queue.
Removes bindings to user provided services for new application
deployments and sets the expected environment variables to `null`.
Listing variables in the manifest allows us to only inject secrets
used by the app. `null` is not a valid value for a variable in a
CloudFoundry manifest, so if any of the keys don't have a match in
the credentials file `cf push` will return an error and stop the
deployment.
specifically, all of the performance platform specific data layout now
happens in performance_platform_client.py - stuff like setting the
_timestamp, period etc, and the perf platform-specific nomenclature is
all handled there.
- TEST and DEV builds set up their own queue lists for reading
- PROD/STAGE/PREVIEW use the separate worker process with the -Q flag
This enables us to rename queues in due course
1) Beat worker now a single instance with 128M memory
2) New worker for periodic tasks. Single worker, 2G ram
3) new worker for priority queues. Standard instance and memory settings.
