diff --git a/Makefile b/Makefile index 3c29ca496..4273f7b7e 100644 --- a/Makefile +++ b/Makefile @@ -12,9 +12,9 @@ GIT_COMMIT ?= $(shell git rev-parse HEAD) .PHONY: bootstrap bootstrap: ## Set up everything to run the app make generate-version-file - poetry install --sync --no-root - poetry self add poetry-dotenv-plugin poetry self update + poetry self add poetry-dotenv-plugin + poetry install --sync --no-root poetry run pre-commit install createdb notification_api || true (poetry run flask db upgrade) || true diff --git a/README.md b/README.md index 8151de6db..c3b283d7b 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,18 @@ # US Notify API -This project is the core of [Notify](https://notifications-admin.app.cloud.gov/). It's cloned from the brilliant work of the team at [GOV.UK Notify](https://github.com/alphagov/notifications-api), cheers! +This project is the core of [Notify](https://notifications-admin.app.cloud.gov/). +It's cloned from the brilliant work of the team at +[GOV.UK Notify](https://github.com/alphagov/notifications-api), cheers! This repo contains: -- A public-facing REST API for Notify, which teams can integrate with using [API clients built by UK](https://www.notifications.service.gov.uk/documentation) -- An internal-only REST API built using Flask to manage services, users, templates, etc., which the [admin UI](http://github.com/18F/notifications-admin) talks to) -- Asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc. +- A public-facing REST API for Notify, which teams can integrate with using + [API clients built by UK](https://www.notifications.service.gov.uk/documentation) +- An internal-only REST API built using Flask to manage services, users, + templates, etc., which the [admin UI](http://github.com/18F/notifications-admin) + talks to) +- Asynchronous workers built using Celery to put things on queues and read them + off to be processed, sent to providers, updated, etc. Our other repositories are: @@ -15,61 +21,310 @@ Our other repositories are: - [us-notify-compliance](https://github.com/GSA/us-notify-compliance/) - [notify-python-demo](https://github.com/GSA/notify-python-demo) -## Local setup +## Before You Start -### Common steps +You will need the following items: -On MacOS, using [Homebrew](https://brew.sh/) for package management is highly recommended. This helps avoid some known installation issues. +- An active cloud.gov account with the correct permissions - speak with your + onboarding buddy for help with + [setting up an account](https://cloud.gov/sign-up/) (requires a `.mil`, + `.gov`, or `.fed.us` email address) and getting access to the + `notify-local-dev` and `notify-staging` spaces. +- Admin priviliges and SSH access on your machine; you may need to work with + your organization's IT support staff if you're not sure or don't currently + have this access. -_Note: If brew is not found: Verify that Homebrew is in your system's PATH. You may need to follow additional instructions to add Homebrew to your PATH in /Users/homefolder/.zprofile:_ +## Local Environment Setup -1. Install pre-requisites for setup: - - [jq](https://stedolan.github.io/jq/): `brew install jq` - - [terraform](https://www.terraform.io/): `brew install terraform` or `brew install tfenv` and use `tfenv` to install `terraform ~> 1.4.0` - - [cf-cli@8](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html): `brew install cloudfoundry/tap/cf-cli@8` - - [postgresql](https://www.postgresql.org/): `brew install postgresql@15` (Homebrew requires a version pin, but any recent version will work) - - [redis](https://redis.io/): `brew install redis` - - [pyenv](https://github.com/pyenv/pyenv): `brew install pyenv` - - [poetry](https://python-poetry.org/docs/#installation): `brew install poetry` -1. [Log into cloud.gov](https://cloud.gov/docs/getting-started/setup/#set-up-the-command-line): `cf login -a api.fr.cloud.gov --sso` -1. Ensure you have access to the `notify-local-dev` and `notify-staging` spaces in cloud.gov -1. Run the development terraform - _(Be sure to clone the repository first and navigate to the terraform directory):_ +This project currently works with these major versions of the following main +components: - ``` - $ cd terraform/development - $ ./run.sh - ``` +- Python 3.9.x +- PostgreSQL 15.x (version 12.x is used in the hosted environments) -1. If you want to send data to New Relic from your local develpment environment, set `NEW_RELIC_LICENSE_KEY` within `.env` -1. Start Postgres && Redis +These instructions will walk you through how to set your machine up with all of +the required tools for this project. - ``` - brew services start postgresql@15 - brew services start redis - ``` +### Project Pre-Requisite Setup -#### Install +On MacOS, using [Homebrew](https://brew.sh/) for package management is highly +recommended. This helps avoid some known installation issues. Start by following +the installation instructions on the Homebrew homepage. -1. Run the project setup (If there are errors, check for Python versions >=3.9,<3.12. Ensure that you have the required database set up and migrations) +**Note:** You will also need Xcode or the Xcode Command Line Tools installed. The +quickest way to do this is is by installing the command line tools in the shell: - `make bootstrap` +```sh +xcode-select –-install +``` -1. Run the web server and background workers +#### Homebrew Setup - `make run-procfile` +If this is your first time installing Homebrew on your machine, you may need to +add its binaries to your system's `$PATH` environment variable so that you can +use the `brew` command. Try running `brew help` to see if Homebrew is +recognized and runs properly. If that fails, then you'll need to add a +configuration line to wherever your `$PATH` environment variable is set. -- Or run them individually: +Your system `$PATH` environment variable is likely set in one of these +locations: -- Run Flask (web server) +For BASH shells: +- `~/.bashrc` +- `~/.bash_profile` +- `~/.profile` - `make run-flask` +For ZSH shells: +- `~/.zshrc` +- `~/.zprofile` -- Run Celery (background worker) +There may be different files that you need to modify for other shell +environments. - `make run-celery` +Which file you need to modify depends on whether or not you are running an +interactive shell or a login shell +(see [this Stack Overflow post](https://stackoverflow.com/questions/18186929/what-are-the-differences-between-a-login-shell-and-interactive-shell) +for an explanation of the differences). If you're still not sure, please ask +the team for help! -### Python dependency management +Once you determine which file you'll need to modify, add these lines before any +lines that add or modify the `$PATH` environment variable; near or at the top +of the file is appropriate: + +```sh +# Homebrew setup +eval "$(/opt/homebrew/bin/brew shellenv)" +``` + +This will make sure Homebrew gets setup correctly. Once you make these changes, +either start a new shell session or source the file +(`source ~/.FILE-YOU-MODIFIED`) you modified to have your system recognize the +changes. + +Verify that Homebrew is now working by trying to run `brew help` again. + +### System-Level Package Installation + +There are several packages you will need to install for your system in order to +get the app running (and these are good to have in general for any software +development). + +Start off with these packages since they're quick and don't require additional +configuration after installation to get working out of the box: + +- [jq](https://stedolan.github.io/jq/) - for working with JSON in the command + line +- [git](https://git-scm.com/) - for version control management +- [tfenv](https://github.com/tfutils/tfenv) - for managing + [Terraform](https://www.terraform.io/) installations +- [cf-cli@8](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html) - for + working with a Cloud Foundry platform (e.g., cloud.gov) +- [redis](https://redis.io/) - required as the backend for the API's + asynchronous job processing +- [vim](https://www.vim.org/) - for editing files more easily in the command + line + +You can install them by running the following: + +```sh +brew install jq git tfenv cloudfoundry/tap/cf-cli@8 redis vim +``` + +#### Terraform Installation + +As a part of the installation above, you just installed `tfenv` to manage +Terraform installations. This is great, but you still need to install Terraform +itself, which can be done with this command: + +```sh +tfenv install latest:^1.4.0 +``` + +_NOTE: This project currently uses the latest `1.4.x release of Terraform._ + +#### Python Installation + +Now we're going to install a tool to help us manage Python versions and +virtual environments on our system. First, we'll install +[pyenv](https://github.com/pyenv/pyenv) and one of its plugins, +[pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv), with Homebrew: + +```sh +brew install pyenv pyenv-virtualenv +``` + +When these finish installing, you'll need to make another adjustment in the +file that you adjusted for your `$PATH` environment variable and Homebrew's +setup. Open the file, and add these lines to it: + +``` +# pyenv setup +export PYENV_ROOT="$HOME/.pyenv" +command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH" +eval "$(pyenv init -)" +eval "$(pyenv virtualenv-init -)" +``` + +Once again, start a new shell session or source the file in your current shell +session to make the changes take effect. + +Now we're ready to install the Python version we need with `pyenv`, like so: + +```sh +pyenv install 3.9 +``` + +This will install the latest version of Python 3.9. + +_NOTE: This project currently runs on Python 3.9.x._ + +#### Python Dependency Installation + +Lastly, we need to install the tool we use to manage Python dependencies within +the project, which is [poetry](https://python-poetry.org/). + +Visit the +[official installer instructions page](https://python-poetry.org/docs/#installing-with-the-official-installer) +and follow the steps to install Poetry directly with the script. + +This will ensure `poetry` doesn't conflict with any project virtual environments +and can update itself properly. + +#### PostgreSQL installation + +We now need to install a database - this project uses PostgreSQL, and Homebrew +requires a version number to be included with it when installing it: + +```sh +brew install postgresql@15 +``` + +_NOTE: This project currently works with PostgreSQL version 15.x; version 12.x is currently used in our hosted environments._ + +You'll now need to modify (or create, if it doesn't already exist) the `$PATH` +environment variable to include the PostgreSQL binaries. Open the file you have +worked with before to adjust your shell environment with the previous steps and +do one of the following: + +If you already have a line that modifies the `$PATH` environment variable, just +add this path into the existing string: + +``` +/opt/homebrew/opt/postgresql@15/bin +``` + +If you don't have a line for your `$PATH` environment variable, add it in like +this, which will include the PostgreSQL binaries: + +``` +export PATH="/opt/homebrew/opt/postgresql@15/bin:$PATH +``` + +_NOTE: You don't want to overwrite your existing `$PATH` environment variable! Hence the reason why it is included on the end like this; paths are separated by a colon._ + +#### Starting PostgreSQL and Redis + +With both PostgreSQL and Redis installed, you now need to start the services. +Run this command so that they're available at all times going forward on your +machine: + +```sh +brew services start postgresql@15 +brew services start redis +``` + +If they're already running, you can run this command instead to make sure the +latest updates are applied to both services: + +```sh +brew services restart postgresql@15 +brew services restart redis +``` + +### First-Time Project Setup + +Once all of pre-requisites for the project are installed and you have a +cloud.gov account, you can now set up the API project and get things running +locally! + +First, clone the respository in the directory of your choosing on your machine: + +```sh +git clone git@github.com:GSA/notifications-api.git +``` + +Now go into the project directory (`notifications-api` by default), create a +virtual environment, and set the local Python version to point to the virtual +environment (assumes version Python `3.9.18` is what is installed on your +machine): + +```sh +cd notifications-api +pyenv virtualenv 3.9.18 notify-api +pyenv local notify-api +``` + +_If you're not sure which version of Python was installed with `pyenv`, you can check by running `pyenv versions` and it'll list everything available currently._ + +Now [log into cloud.gov](https://cloud.gov/docs/getting-started/setup/#set-up-the-command-line) +in the command line by using this command: + +```sh +cf login -a api.fr.cloud.gov --sso +``` + +_REMINDER: Ensure you have access to the `notify-local-dev` and `notify-staging` spaces in cloud.gov_ + +Now run the development Terraform setup by navigating to the development +folder and running the script in it: + +```sh +cd terraform/development +./run.sh +``` + +In addition to some infrastructure setup, this will also create a local `.env` +file for you in the project's root directory, which will include a handful of +project-specific environment variables. + +Lastly, if you didn't already start PostgreSQL and Redis above, be sure to do +so now: + +```sh +brew services start postgresql@15 +brew services start redis +``` + +## Running the Project and Routine Maintenance + +The first time you run the project you'll need to run the project setup from the +root project directory: + +```sh +make bootstrap +``` + +This command is handled by the `Makefile` file in the root project directory, as +are a few others. + +_NOTE: You'll want to occasionally run `make bootstrap` to keep your project up-to-date, especially when there are dependency updates._ + +Now you can run the web server and background workers for asynchronous jobs: + +```sh +make run-procfile +``` + +This will run all of the services within the same shell session. If you need to +run them separately to help with debugging or tracing logs, you can do so by +opening three sepearate shell sessions and running one of these commands in each +one separately: + +- `make run-celery` - Handles the asynchronous jobs +- `make run-celery-beat` - Handles the scheduling of asynchronous jobs +- `make run-flask` - Runs the web server + +## Python Dependency Management We're using [`Poetry`](https://python-poetry.org/) for managing our Python dependencies and local virtual environments. When it comes to managing the @@ -83,7 +338,7 @@ that have not been fully tested with the project yet. If you're just trying to update a dependency to a newer (or the latest) version, you should let Poetry take care of that for you by running the following: -``` +```sh poetry update [...] ``` @@ -98,7 +353,7 @@ In either situation, once you are finished and have verified the dependency changes are working, please be sure to commit both the `pyproject.toml` and `poetry.lock` files. -### Keeping the notification-utils dependency up-to-date +### Keeping the notification-utils Dependency Up-to-Date The `notifications-utils` dependency references the other repository we have at https://github.com/GSA/notifications-utils - this dependency requires a bit of @@ -115,15 +370,27 @@ You can do this by going through these steps: - Make a new PR with the change - Have the PR get reviewed and merged -### Known installation issues +## Known Installation Issues + +### Python Installation Errors On M1 Macs, if you get a `fatal error: 'Python.h' file not found` message, try a -different method of installing Python. Installation via `pyenv` is known to -work. +different method of installing Python. The recommended approach is to use +[`pyenv`](https://github.com/pyenv/pyenv), as noted above in the installation +instructions. + +If you're using PyCharm for Python development, we've noticed some quirkiness +with the IDE and the interaction between Poetry and virtual environment +management that could cause a variety of problems to come up during project +setup and dependency management. Other tools, such as Visual Studio Code, have +proven to be a smoother experience for folks. + +### PostgreSQL Installation Errors A direct installation of PostgreSQL will not put the `createdb` command on your `$PATH`. It can be added there in your shell startup script, or a -Homebrew-managed installation of PostgreSQL will take care of it. +Homebrew-managed installation of PostgreSQL will take care of it. See the +instructions above for more details. ## Documentation diff --git a/sample.env b/sample.env index 9cd957e64..f53559acb 100644 --- a/sample.env +++ b/sample.env @@ -39,13 +39,14 @@ NOTIFY_APP_NAME=api ############################################################# +# E2E Testing + +NOTIFY_E2E_TEST_EMAIL=example@fake.gov +NOTIFY_E2E_TEST_PASSWORD="don't write secrets to the sample file" + +############################################################# + # Flask FLASK_APP=application.py FLASK_DEBUG=true WERKZEUG_DEBUG_PIN=off - -############################################################# - -# New Relic -NEW_RELIC_CONFIG_FILE=newrelic.ini -NEW_RELIC_LICENSE_KEY="don't write secrets to the sample file"