scripts/run_with_docker.sh

#!/bin/bash
set -eu

DOCKER_IMAGE_NAME=notifications-api

source environment.sh

# this script should be run from within your virtualenv so you can access the aws cli
AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-"$(aws configure get aws_access_key_id)"}
AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-"$(aws configure get aws_secret_access_key)"}
SQLALCHEMY_DATABASE_URI="postgresql://postgres@host.docker.internal/notification_api"
REDIS_URL="redis://host.docker.internal:6379"
API_HOST_NAME="http://host.docker.internal:6011"

docker run -it --rm \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -e SQLALCHEMY_DATABASE_URI=$SQLALCHEMY_DATABASE_URI \
  -e REDIS_ENABLED=${REDIS_ENABLED:-0} \
  -e REDIS_URL=$REDIS_URL \
  -e API_HOST_NAME=$API_HOST_NAME \
  -v $(pwd):/home/vcap/app \
  ${DOCKER_IMAGE_NAME} \
  ${@}
add script to run celery from within docker as a team we primarily develop locally. However, we've been experiencing issues with pycurl, a subdependency of celery, that is notoriously difficult to install on mac. On top of the existing issues, we're also seeing it conflict with pyproj in bizarre ways (where the order of imports between pyproj and pycurl result in different configurations of dynamically linked C libraries being loaded. You are encouraged to attempt to install pycurl locally, following these instructions: https://github.com/alphagov/notifications-manuals/wiki/Getting-Started#pycurl However, if you aren't having any luck, you can instead now run celery in a docker container. `make run-celery-with-docker` This will build a container, install the dependencies, and run celery (with the default of four concurrent workers). It will pull aws variables from your aws configuration as boto would normally, and it will attempt to connect to your local database with the user `postgres`. If your local database is configured differently (for example, with a different user, or on a different port), then you can set the SQLALCHEMY_DATABASE_URI locally to override that. 2022-01-31 18:35:39 +00:00			`#!/bin/bash`
move environment variables to their own lines and set -eu this means that if the environment variable can't be set (for example, if you don't have aws-cli installed) then there's a suitable error message early on. 2022-02-01 14:19:52 +00:00			`set -eu`

add script to run celery from within docker as a team we primarily develop locally. However, we've been experiencing issues with pycurl, a subdependency of celery, that is notoriously difficult to install on mac. On top of the existing issues, we're also seeing it conflict with pyproj in bizarre ways (where the order of imports between pyproj and pycurl result in different configurations of dynamically linked C libraries being loaded. You are encouraged to attempt to install pycurl locally, following these instructions: https://github.com/alphagov/notifications-manuals/wiki/Getting-Started#pycurl However, if you aren't having any luck, you can instead now run celery in a docker container. `make run-celery-with-docker` This will build a container, install the dependencies, and run celery (with the default of four concurrent workers). It will pull aws variables from your aws configuration as boto would normally, and it will attempt to connect to your local database with the user `postgres`. If your local database is configured differently (for example, with a different user, or on a different port), then you can set the SQLALCHEMY_DATABASE_URI locally to override that. 2022-01-31 18:35:39 +00:00			`DOCKER_IMAGE_NAME=notifications-api`

			`source environment.sh`

move environment variables to their own lines and set -eu this means that if the environment variable can't be set (for example, if you don't have aws-cli installed) then there's a suitable error message early on. 2022-02-01 14:19:52 +00:00			`# this script should be run from within your virtualenv so you can access the aws cli`
			`AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-"$(aws configure get aws_access_key_id)"}`
			`AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-"$(aws configure get aws_secret_access_key)"}`
Fix letter functional tests to work in Docker Currently "test_send_letter_notification_via_api" fails at the final stage in create-fake-letter-response-file [^1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=6011): Max retries exceeded with url: /notifications/letter/dvla (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff95ffc460>: Failed to establish a new connection: [Errno 111] Connection refused')) This only applies when running in Docker so the default should still be "localhost" for the Flask app itself. [^1]: https://github.com/alphagov/notifications-api/blob/5093064533e558d48add2c373eb5eeaf752e6b8e/app/celery/research_mode_tasks.py#L57 2022-03-09 11:07:40 +00:00			`SQLALCHEMY_DATABASE_URI="postgresql://postgres@host.docker.internal/notification_api"`
Minor tweaks in response to PR comments 2022-02-25 17:49:07 +00:00			`REDIS_URL="redis://host.docker.internal:6379"`
Fix letter functional tests to work in Docker Currently "test_send_letter_notification_via_api" fails at the final stage in create-fake-letter-response-file [^1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=6011): Max retries exceeded with url: /notifications/letter/dvla (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff95ffc460>: Failed to establish a new connection: [Errno 111] Connection refused')) This only applies when running in Docker so the default should still be "localhost" for the Flask app itself. [^1]: https://github.com/alphagov/notifications-api/blob/5093064533e558d48add2c373eb5eeaf752e6b8e/app/celery/research_mode_tasks.py#L57 2022-03-09 11:07:40 +00:00			`API_HOST_NAME="http://host.docker.internal:6011"`
move environment variables to their own lines and set -eu this means that if the environment variable can't be set (for example, if you don't have aws-cli installed) then there's a suitable error message early on. 2022-02-01 14:19:52 +00:00
add script to run celery from within docker as a team we primarily develop locally. However, we've been experiencing issues with pycurl, a subdependency of celery, that is notoriously difficult to install on mac. On top of the existing issues, we're also seeing it conflict with pyproj in bizarre ways (where the order of imports between pyproj and pycurl result in different configurations of dynamically linked C libraries being loaded. You are encouraged to attempt to install pycurl locally, following these instructions: https://github.com/alphagov/notifications-manuals/wiki/Getting-Started#pycurl However, if you aren't having any luck, you can instead now run celery in a docker container. `make run-celery-with-docker` This will build a container, install the dependencies, and run celery (with the default of four concurrent workers). It will pull aws variables from your aws configuration as boto would normally, and it will attempt to connect to your local database with the user `postgres`. If your local database is configured differently (for example, with a different user, or on a different port), then you can set the SQLALCHEMY_DATABASE_URI locally to override that. 2022-01-31 18:35:39 +00:00			`docker run -it --rm \`
move environment variables to their own lines and set -eu this means that if the environment variable can't be set (for example, if you don't have aws-cli installed) then there's a suitable error message early on. 2022-02-01 14:19:52 +00:00			`-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \`
			`-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \`
			`-e SQLALCHEMY_DATABASE_URI=$SQLALCHEMY_DATABASE_URI \`
Iterate local development with Docker This makes a few changes to: - Make local development consistent with our other apps. It's now faster to start Celery locally since we don't try to build the image each time - this is usually quick, but unnecessary. - Add support for connecting to a local Redis instance. Note that the previous suggestion of "REDIS = True" was incorrect as this would be turned into the literal string "True". I've also co-located and extended the recipes in the Makefile to make them a bit more visible. 2022-02-24 17:11:46 +00:00			`-e REDIS_ENABLED=${REDIS_ENABLED:-0} \`
Minor tweaks in response to PR comments 2022-02-25 17:49:07 +00:00			`-e REDIS_URL=$REDIS_URL \`
Fix letter functional tests to work in Docker Currently "test_send_letter_notification_via_api" fails at the final stage in create-fake-letter-response-file [^1]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=6011): Max retries exceeded with url: /notifications/letter/dvla (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff95ffc460>: Failed to establish a new connection: [Errno 111] Connection refused')) This only applies when running in Docker so the default should still be "localhost" for the Flask app itself. [^1]: https://github.com/alphagov/notifications-api/blob/5093064533e558d48add2c373eb5eeaf752e6b8e/app/celery/research_mode_tasks.py#L57 2022-03-09 11:07:40 +00:00			`-e API_HOST_NAME=$API_HOST_NAME \`
add script to run celery from within docker as a team we primarily develop locally. However, we've been experiencing issues with pycurl, a subdependency of celery, that is notoriously difficult to install on mac. On top of the existing issues, we're also seeing it conflict with pyproj in bizarre ways (where the order of imports between pyproj and pycurl result in different configurations of dynamically linked C libraries being loaded. You are encouraged to attempt to install pycurl locally, following these instructions: https://github.com/alphagov/notifications-manuals/wiki/Getting-Started#pycurl However, if you aren't having any luck, you can instead now run celery in a docker container. `make run-celery-with-docker` This will build a container, install the dependencies, and run celery (with the default of four concurrent workers). It will pull aws variables from your aws configuration as boto would normally, and it will attempt to connect to your local database with the user `postgres`. If your local database is configured differently (for example, with a different user, or on a different port), then you can set the SQLALCHEMY_DATABASE_URI locally to override that. 2022-01-31 18:35:39 +00:00			`-v $(pwd):/home/vcap/app \`
			`${DOCKER_IMAGE_NAME} \`
			`${@}`