Commit Graph

10 Commits

Author SHA1 Message Date
Alexey Bezhan
676e3ec39a Stream delivery worker logs to stdout when running on PaaS
Our application servers and celery workers write logs both to a
file that is shipped to CloudWatch and to stdout, which is picked
up by CloudFoundry and sent to Logit Logstash.

This works with gunicorn and single-worker celery deployments, however
celery multi daemonizes worker processes, which detaches them from
stdout, so there's no log output in `cf logs` or Logit.

To fix this, we start a separate tail process to duplicate logs written
to a file to stdout, which should be picked up by CloudFoundry.
2018-06-29 11:49:02 +01:00
Athanasios Voutsadakis
d365921093 Address PR comments
Make timeout 9s on both files
Use more descriptive variable name
Declare the logs dir as a constant
2018-03-06 16:11:42 +00:00
Athanasios Voutsadakis
3045f6233b Use continue instead of break on SIGKILL
`break`ing would keep trying to kill the same process and never move to
the next
2018-02-28 14:50:23 +00:00
Athanasios Voutsadakis
b158c66705 Set timeout to 9
This will allow us an extra second to `kill -9` any remaining processes.
2018-02-28 14:37:02 +00:00
Athanasios Voutsadakis
7d562f9e85 Get empty array when no .pid file exists 2018-02-28 14:36:40 +00:00
Athanasios Voutsadakis
1adee47230 Always get the latest PIDs to check 2018-02-28 14:36:06 +00:00
Athanasios Voutsadakis
138d4eee25 Better logging of PIDs array 2018-02-28 14:34:48 +00:00
Athanasios Voutsadakis
bef80e3414 Use eval to run the command instead of exec
`exec` is replacing the current shell to run the command, which means that
the script execution stops at that line.

Passing it to the background with `exec "$@" &` won't work either,
because the script will move directly to the next command where it
looks for the `.pid` files that have not yet been created because celery
takes a few seconds to spin up all the processes.

Using `sleep X` to remedy this seems just wrong given that

1. we can use `eval` that blocks until the command returns
2. there is no obvious benefit in sticking with `exec`
2018-02-28 14:16:15 +00:00
Athanasios Voutsadakis
9a8e771d95 Lower the termination timeout to 10s
Cloudfoundry only gives us 10s to terminate our processes:
https://docs.cloudfoundry.org/devguide/deploy-apps/app-lifecycle.html#shutdown
2018-02-27 17:44:16 +00:00
Athanasios Voutsadakis
7c36c3894d Add a start script for running celery multi
The existing script would not work with `celery multi` as it was trying
to put it in the background and the get its pid.~

`celery multi` creates a number of worker processes and stores their
PIDs in files named celeryN.pid, where N the index number of the worker
(starting at 1).
2018-02-27 17:37:09 +00:00