darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2026-07-18 13:40:31 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
953a545aea19fcba93342d1349ee8df94de77d3e
notifications-api/manifest-delivery-base.yml

101 lines
3.3 KiB
YAML
Raw Normal View History

Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
---
buildpack: python_buildpack
health-check-type: none
no-route: true
services:
- notify-db
instances: 1
Increase memory for PaaS
2017-03-06 14:48:27 +00:00
memory: 1G
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
Replace manifest services with lists of env variables Removes bindings to user provided services for new application deployments and sets the expected environment variables to `null`. Listing variables in the manifest allows us to only inject secrets used by the app. `null` is not a valid value for a variable in a CloudFoundry manifest, so if any of the keys don't have a match in the credentials file `cf push` will return an error and stop the deployment.
2018-01-05 14:27:17 +00:00
env:
# Credentials variables
ADMIN_BASE_URL: null
ADMIN_CLIENT_SECRET: null
API_HOST_NAME: null
DANGEROUS_SALT: null
SECRET_KEY: null
ROUTE_SECRET_KEY_1: null
ROUTE_SECRET_KEY_2: null
PERFORMANCE_PLATFORM_ENDPOINTS: null
NOTIFICATION_QUEUE_PREFIX: null
AWS_ACCESS_KEY_ID: null
AWS_SECRET_ACCESS_KEY: null
STATSD_PREFIX: null
Add utils DeskproClient and configuration variables Deskpro client is used to create tickets from celery alerting tasks (eg alerts for missing ack or response files from DVLA).
2018-01-17 14:36:02 +00:00
DESKPRO_API_KEY: null
DESKPRO_API_HOST: null
Replace manifest services with lists of env variables Removes bindings to user provided services for new application deployments and sets the expected environment variables to `null`. Listing variables in the manifest allows us to only inject secrets used by the app. `null` is not a valid value for a variable in a CloudFoundry manifest, so if any of the keys don't have a match in the credentials file `cf push` will return an error and stop the deployment.
2018-01-05 14:27:17 +00:00
MMG_URL: null
MMG_API_KEY: null
MMG_INBOUND_SMS_AUTH: null
MMG_INBOUND_SMS_USERNAME: null
FIRETEXT_API_KEY: null
LOADTESTING_API_KEY: null
FIRETEXT_INBOUND_SMS_AUTH: null
REDIS_ENABLED: null
REDIS_URL: null
TEMPLATE_PREVIEW_API_HOST: null
TEMPLATE_PREVIEW_API_KEY: null
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
applications:
- name: notify-delivery-celery-beat
remove aws_run_celery file it's no longer relevant since the switch to PaaS
2017-07-31 11:12:43 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery beat --loglevel=INFO
A few changes to the celery workers 1) Beat worker now a single instance with 128M memory 2) New worker for periodic tasks. Single worker, 2G ram 3) new worker for priority queues. Standard instance and memory settings.
2017-04-04 13:43:28 +01:00
instances: 1
memory: 128M
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
env:
NOTIFY_APP_NAME: delivery-celery-beat
- name: notify-delivery-worker-database
remove aws_run_celery file it's no longer relevant since the switch to PaaS
2017-07-31 11:12:43 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Q database-tasks
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
env:
NOTIFY_APP_NAME: delivery-worker-database
- name: notify-delivery-worker-research
remove aws_run_celery file it's no longer relevant since the switch to PaaS
2017-07-31 11:12:43 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=5 -Q research-mode-tasks
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
env:
NOTIFY_APP_NAME: delivery-worker-research
- name: notify-delivery-worker-sender
remove SEND_COMBINED
2017-07-31 13:29:30 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Q send-sms-tasks,send-email-tasks
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
env:
NOTIFY_APP_NAME: delivery-worker-sender
Broke the periodic queue out from the standard worker. - now has own worker with worker count of 2 - this is too ensure that any issues with scheduled jobs do not affect core applications
2017-04-04 11:23:54 +01:00
- name: notify-delivery-worker-periodic
remove aws_run_celery file it's no longer relevant since the switch to PaaS
2017-07-31 11:12:43 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=2 -Q periodic-tasks,statistics-tasks
A few changes to the celery workers 1) Beat worker now a single instance with 128M memory 2) New worker for periodic tasks. Single worker, 2G ram 3) new worker for priority queues. Standard instance and memory settings.
2017-04-04 13:43:28 +01:00
instances: 1
Broke the periodic queue out from the standard worker. - now has own worker with worker count of 2 - this is too ensure that any issues with scheduled jobs do not affect core applications
2017-04-04 11:23:54 +01:00
env:
A few changes to the celery workers 1) Beat worker now a single instance with 128M memory 2) New worker for periodic tasks. Single worker, 2G ram 3) new worker for priority queues. Standard instance and memory settings.
2017-04-04 13:43:28 +01:00
NOTIFY_APP_NAME: delivery-worker-periodic
- name: notify-delivery-worker-priority
remove aws_run_celery file it's no longer relevant since the switch to PaaS
2017-07-31 11:12:43 +01:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=5 -Q priority-tasks
A few changes to the celery workers 1) Beat worker now a single instance with 128M memory 2) New worker for periodic tasks. Single worker, 2G ram 3) new worker for priority queues. Standard instance and memory settings.
2017-04-04 13:43:28 +01:00
env:
NOTIFY_APP_NAME: delivery-worker-priority
Broke the periodic queue out from the standard worker. - now has own worker with worker count of 2 - this is too ensure that any issues with scheduled jobs do not affect core applications
2017-04-04 11:23:54 +01:00
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
- name: notify-delivery-worker
Tasks that are put on the notify-internal-tasks queue are notifications sent by the notify service, including 2FA and email auth messages. These should never be delayed by other messages in the queue. Created a separate delivery app to consume the notify-internal-tasks queue and only that queue.
2018-01-24 17:12:26 +00:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Q job-tasks,retry-tasks,create-letters-pdf-tasks,letter-tasks
Add awslogs logging for PaaS
2017-02-13 18:04:08 +00:00
env:
NOTIFY_APP_NAME: delivery-worker
Add worker-receipts app
2017-10-27 12:05:15 +01:00
Tasks that are put on the notify-internal-tasks queue are notifications sent by the notify service, including 2FA and email auth messages. These should never be delayed by other messages in the queue. Created a separate delivery app to consume the notify-internal-tasks queue and only that queue.
2018-01-24 17:12:26 +00:00
# Only consume the notify-internal-tasks queue on this app so that Notify messages are processed as a priority
- name: notify-delivery-worker-internal
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Q notify-internal-tasks
env:
NOTIFY_APP_NAME: delivery-worker-internal
Add worker-receipts app
2017-10-27 12:05:15 +01:00
- name: notify-delivery-worker-receipts
When there are high volumes of emails being processed the SES callback task will trigger the service-callback task. This PR creates a new worker to deal with the service-callback so that one event doesn't block the other.
2018-02-09 09:39:14 +00:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO --concurrency=11 -Q ses-callbacks
Add worker-receipts app
2017-10-27 12:05:15 +01:00
env:
NOTIFY_APP_NAME: delivery-worker-receipts
When there are high volumes of emails being processed the SES callback task will trigger the service-callback task. This PR creates a new worker to deal with the service-callback so that one event doesn't block the other.
2018-02-09 09:39:14 +00:00
- name: notify-delivery-worker-service-callbacks
Switch service callback workers to use eventlet pool implementation Service callbacks are I/O bound and can take a long time if the callback URL takes a long time to respond. This is a perfect use case for an eventlet worker pool since it allows spawning multiple green threads (1000 in the proposed configuration) to execute HTTP requests concurrently without a significant increase in CPU load or memory usage needed for adding additional worker processes.
2018-02-12 16:19:23 +00:00
command: scripts/run_app_paas.sh celery -A run_celery.notify_celery worker --loglevel=INFO -P eventlet -c 1000 -Q service-callbacks
When there are high volumes of emails being processed the SES callback task will trigger the service-callback task. This PR creates a new worker to deal with the service-callback so that one event doesn't block the other.
2018-02-09 09:39:14 +00:00
env:
Switch service callback workers to use eventlet pool implementation Service callbacks are I/O bound and can take a long time if the callback URL takes a long time to respond. This is a perfect use case for an eventlet worker pool since it allows spawning multiple green threads (1000 in the proposed configuration) to execute HTTP requests concurrently without a significant increase in CPU load or memory usage needed for adding additional worker processes.
2018-02-12 16:19:23 +00:00
NOTIFY_APP_NAME: delivery-worker-service-callbacks
Reference in New Issue Copy Permalink