Merge pull request #597 from GSA/update-project-setup

Update README with new instructions and improve setup

Makefile

@@ -12,9 +12,9 @@ GIT_COMMIT ?= $(shell git rev-parse HEAD)
 .PHONY: bootstrap
 bootstrap: ## Set up everything to run the app
 	make generate-version-file
-	poetry install --sync --no-root
-	poetry self add poetry-dotenv-plugin
 	poetry self update
+	poetry self add poetry-dotenv-plugin
+	poetry install --sync --no-root
 	poetry run pre-commit install
 	createdb notification_api || true
 	(poetry run flask db upgrade) || true

README.md

@@ -1,12 +1,18 @@
 # US Notify API
 
-This project is the core of [Notify](https://notifications-admin.app.cloud.gov/). It's cloned from the brilliant work of the team at [GOV.UK Notify](https://github.com/alphagov/notifications-api), cheers!
+This project is the core of [Notify](https://notifications-admin.app.cloud.gov/).
+It's cloned from the brilliant work of the team at
+[GOV.UK Notify](https://github.com/alphagov/notifications-api), cheers!
 
 This repo contains:
 
-- A public-facing REST API for Notify, which teams can integrate with using [API clients built by UK](https://www.notifications.service.gov.uk/documentation)
+- A public-facing REST API for Notify, which teams can integrate with using
-- An internal-only REST API built using Flask to manage services, users, templates, etc., which the [admin UI](http://github.com/18F/notifications-admin) talks to)
+ [API clients built by UK](https://www.notifications.service.gov.uk/documentation)
-- Asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc.
+- An internal-only REST API built using Flask to manage services, users,
+  templates, etc., which the [admin UI](http://github.com/18F/notifications-admin)
+  talks to)
+- Asynchronous workers built using Celery to put things on queues and read them
+  off to be processed, sent to providers, updated, etc.
 
 Our other repositories are:
 
@@ -15,61 +21,310 @@ Our other repositories are:
 - [us-notify-compliance](https://github.com/GSA/us-notify-compliance/)
 - [notify-python-demo](https://github.com/GSA/notify-python-demo)
 
-## Local setup
+## Before You Start
 
-### Common steps
+You will need the following items:
 
-On MacOS, using [Homebrew](https://brew.sh/) for package management is highly recommended. This helps avoid some known installation issues.
+- An active cloud.gov account with the correct permissions - speak with your
+  onboarding buddy for help with
+  [setting up an account](https://cloud.gov/sign-up/) (requires a `.mil`,
+  `.gov`, or `.fed.us` email address) and getting access to the
+  `notify-local-dev` and `notify-staging` spaces.
+- Admin priviliges and SSH access on your machine; you may need to work with
+  your organization's IT support staff if you're not sure or don't currently
+  have this access.
 
-_Note: If brew is not found: Verify that Homebrew is in your system's PATH. You may need to follow additional instructions to add Homebrew to your PATH in /Users/homefolder/.zprofile:_
+## Local Environment Setup
 
-1.  Install pre-requisites for setup:
+This project currently works with these major versions of the following main
-    - [jq](https://stedolan.github.io/jq/): `brew install jq`
+components:
-    - [terraform](https://www.terraform.io/): `brew install terraform` or `brew install tfenv` and use `tfenv` to install `terraform ~> 1.4.0`
-    - [cf-cli@8](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html): `brew install cloudfoundry/tap/cf-cli@8`
-    - [postgresql](https://www.postgresql.org/): `brew install postgresql@15` (Homebrew requires a version pin, but any recent version will work)
-    - [redis](https://redis.io/): `brew install redis`
-    - [pyenv](https://github.com/pyenv/pyenv): `brew install pyenv`
-    - [poetry](https://python-poetry.org/docs/#installation): `brew install poetry`
-1.  [Log into cloud.gov](https://cloud.gov/docs/getting-started/setup/#set-up-the-command-line): `cf login -a api.fr.cloud.gov --sso`
-1.  Ensure you have access to the `notify-local-dev` and `notify-staging` spaces in cloud.gov
-1.  Run the development terraform
-    _(Be sure to clone the repository first and navigate to the terraform directory):_
 
-            ```
+- Python 3.9.x
-            $ cd terraform/development
+- PostgreSQL 15.x (version 12.x is used in the hosted environments)
-            $ ./run.sh
-            ```
 
-1.  If you want to send data to New Relic from your local develpment environment, set `NEW_RELIC_LICENSE_KEY` within `.env`
+These instructions will walk you through how to set your machine up with all of
-1.  Start Postgres && Redis
+the required tools for this project.
 
-         ```
+### Project Pre-Requisite Setup
-         brew services start postgresql@15
-         brew services start redis
-         ```
 
-#### Install
+On MacOS, using [Homebrew](https://brew.sh/) for package management is highly
+recommended. This helps avoid some known installation issues. Start by following
+the installation instructions on the Homebrew homepage.
 
-1. Run the project setup (If there are errors, check for Python versions >=3.9,<3.12. Ensure that you have the required database set up and migrations)
+**Note:** You will also need Xcode or the Xcode Command Line Tools installed. The
+quickest way to do this is is by installing the command line tools in the shell:
 
-   `make bootstrap`
+```sh
+xcode-select –-install
+```
 
-1. Run the web server and background workers
+#### Homebrew Setup
 
-   `make run-procfile`
+If this is your first time installing Homebrew on your machine, you may need to
+add its binaries to your system's `$PATH` environment variable so that you can
+use the `brew` command. Try running `brew help` to see if Homebrew is
+recognized and runs properly. If that fails, then you'll need to add a
+configuration line to wherever your `$PATH` environment variable is set.
 
-- Or run them individually:
+Your system `$PATH` environment variable is likely set in one of these
+locations:
 
-- Run Flask (web server)
+For BASH shells:
+- `~/.bashrc`
+- `~/.bash_profile`
+- `~/.profile`
 
-  `make run-flask`
+For ZSH shells:
+- `~/.zshrc`
+- `~/.zprofile`
 
-- Run Celery (background worker)
+There may be different files that you need to modify for other shell
+environments.
 
-  `make run-celery`
+Which file you need to modify depends on whether or not you are running an
+interactive shell or a login shell
+(see [this Stack Overflow post](https://stackoverflow.com/questions/18186929/what-are-the-differences-between-a-login-shell-and-interactive-shell)
+for an explanation of the differences).  If you're still not sure, please ask
+the team for help!
 
-### Python dependency management
+Once you determine which file you'll need to modify, add these lines before any
+lines that add or modify the `$PATH` environment variable; near or at the top
+of the file is appropriate:
+
+```sh
+# Homebrew setup
+eval "$(/opt/homebrew/bin/brew shellenv)"
+```
+
+This will make sure Homebrew gets setup correctly. Once you make these changes,
+either start a new shell session or source the file
+(`source ~/.FILE-YOU-MODIFIED`) you modified to have your system recognize the
+changes.
+
+Verify that Homebrew is now working by trying to run `brew help` again.
+
+### System-Level Package Installation
+
+There are several packages you will need to install for your system in order to
+get the app running (and these are good to have in general for any software
+development).
+
+Start off with these packages since they're quick and don't require additional
+configuration after installation to get working out of the box:
+
+- [jq](https://stedolan.github.io/jq/) - for working with JSON in the command
+  line
+- [git](https://git-scm.com/) - for version control management
+- [tfenv](https://github.com/tfutils/tfenv) - for managing
+  [Terraform](https://www.terraform.io/) installations
+- [cf-cli@8](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html) - for
+  working with a Cloud Foundry platform (e.g., cloud.gov)
+- [redis](https://redis.io/) - required as the backend for the API's
+  asynchronous job processing
+- [vim](https://www.vim.org/) - for editing files more easily in the command
+  line
+
+You can install them by running the following:
+
+```sh
+brew install jq git tfenv cloudfoundry/tap/cf-cli@8 redis vim
+```
+
+#### Terraform Installation
+
+As a part of the installation above, you just installed `tfenv` to manage
+Terraform installations. This is great, but you still need to install Terraform
+itself, which can be done with this command:
+
+```sh
+tfenv install latest:^1.4.0
+```
+
+_NOTE: This project currently uses the latest `1.4.x release of Terraform._
+
+#### Python Installation
+
+Now we're going to install a tool to help us manage Python versions and
+virtual environments on our system.  First, we'll install
+[pyenv](https://github.com/pyenv/pyenv) and one of its plugins,
+[pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv), with Homebrew:
+
+```sh
+brew install pyenv pyenv-virtualenv
+```
+
+When these finish installing, you'll need to make another adjustment in the
+file that you adjusted for your `$PATH` environment variable and Homebrew's
+setup. Open the file, and add these lines to it:
+
+```
+# pyenv setup
+export PYENV_ROOT="$HOME/.pyenv"
+command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
+eval "$(pyenv init -)"
+eval "$(pyenv virtualenv-init -)"
+```
+
+Once again, start a new shell session or source the file in your current shell
+session to make the changes take effect.
+
+Now we're ready to install the Python version we need with `pyenv`, like so:
+
+```sh
+pyenv install 3.9
+```
+
+This will install the latest version of Python 3.9.
+
+_NOTE: This project currently runs on Python 3.9.x._
+
+#### Python Dependency Installation
+
+Lastly, we need to install the tool we use to manage Python dependencies within
+the project, which is [poetry](https://python-poetry.org/).
+
+Visit the
+[official installer instructions page](https://python-poetry.org/docs/#installing-with-the-official-installer)
+and follow the steps to install Poetry directly with the script.
+
+This will ensure `poetry` doesn't conflict with any project virtual environments
+and can update itself properly.
+
+#### PostgreSQL installation
+
+We now need to install a database - this project uses PostgreSQL, and Homebrew
+requires a version number to be included with it when installing it:
+
+```sh
+brew install postgresql@15
+```
+
+_NOTE: This project currently works with PostgreSQL version 15.x; version 12.x is currently used in our hosted environments._
+
+You'll now need to modify (or create, if it doesn't already exist) the `$PATH`
+environment variable to include the PostgreSQL binaries. Open the file you have
+worked with before to adjust your shell environment with the previous steps and
+do one of the following:
+
+If you already have a line that modifies the `$PATH` environment variable, just
+add this path into the existing string:
+
+```
+/opt/homebrew/opt/postgresql@15/bin
+```
+
+If you don't have a line for your `$PATH` environment variable, add it in like
+this, which will include the PostgreSQL binaries:
+
+```
+export PATH="/opt/homebrew/opt/postgresql@15/bin:$PATH
+```
+
+_NOTE: You don't want to overwrite your existing `$PATH` environment variable! Hence the reason why it is included on the end like this; paths are separated by a colon._
+
+#### Starting PostgreSQL and Redis
+
+With both PostgreSQL and Redis installed, you now need to start the services.
+Run this command so that they're available at all times going forward on your
+machine:
+
+```sh
+brew services start postgresql@15
+brew services start redis
+```
+
+If they're already running, you can run this command instead to make sure the
+latest updates are applied to both services:
+
+```sh
+brew services restart postgresql@15
+brew services restart redis
+```
+
+### First-Time Project Setup
+
+Once all of pre-requisites for the project are installed and you have a
+cloud.gov account, you can now set up the API project and get things running
+locally!
+
+First, clone the respository in the directory of your choosing on your machine:
+
+```sh
+git clone git@github.com:GSA/notifications-api.git
+```
+
+Now go into the project directory (`notifications-api` by default), create a
+virtual environment, and set the local Python version to point to the virtual
+environment (assumes version Python `3.9.18` is what is installed on your
+machine):
+
+```sh
+cd notifications-api
+pyenv virtualenv 3.9.18 notify-api
+pyenv local notify-api
+```
+
+_If you're not sure which version of Python was installed with `pyenv`, you can check by running `pyenv versions` and it'll list everything available currently._
+
+Now [log into cloud.gov](https://cloud.gov/docs/getting-started/setup/#set-up-the-command-line)
+in the command line by using this command:
+
+```sh
+cf login -a api.fr.cloud.gov --sso
+```
+
+_REMINDER: Ensure you have access to the `notify-local-dev` and `notify-staging` spaces in cloud.gov_
+
+Now run the development Terraform setup by navigating to the development
+folder and running the script in it:
+
+```sh
+cd terraform/development
+./run.sh
+```
+
+In addition to some infrastructure setup, this will also create a local `.env`
+file for you in the project's root directory, which will include a handful of
+project-specific environment variables.
+
+Lastly, if you didn't already start PostgreSQL and Redis above, be sure to do
+so now:
+
+```sh
+brew services start postgresql@15
+brew services start redis
+```
+
+## Running the Project and Routine Maintenance
+
+The first time you run the project you'll need to run the project setup from the
+root project directory:
+
+```sh
+make bootstrap
+```
+
+This command is handled by the `Makefile` file in the root project directory, as
+are a few others.
+
+_NOTE: You'll want to occasionally run `make bootstrap` to keep your project up-to-date, especially when there are dependency updates._
+
+Now you can run the web server and background workers for asynchronous jobs:
+
+```sh
+make run-procfile
+```
+
+This will run all of the services within the same shell session.  If you need to
+run them separately to help with debugging or tracing logs, you can do so by
+opening three sepearate shell sessions and running one of these commands in each
+one separately:
+
+- `make run-celery` - Handles the asynchronous jobs
+- `make run-celery-beat` - Handles the scheduling of asynchronous jobs
+- `make run-flask` - Runs the web server
+
+## Python Dependency Management
 
 We're using [`Poetry`](https://python-poetry.org/) for managing our Python
 dependencies and local virtual environments. When it comes to managing the
@@ -83,7 +338,7 @@ that have not been fully tested with the project yet.
 If you're just trying to update a dependency to a newer (or the latest) version,
 you should let Poetry take care of that for you by running the following:
 
-```
+```sh
 poetry update <dependency> [<dependency>...]
 ```
 
@@ -98,7 +353,7 @@ In either situation, once you are finished and have verified the dependency
 changes are working, please be sure to commit both the `pyproject.toml` and
 `poetry.lock` files.
 
-### Keeping the notification-utils dependency up-to-date
+### Keeping the notification-utils Dependency Up-to-Date
 
 The `notifications-utils` dependency references the other repository we have at
 https://github.com/GSA/notifications-utils - this dependency requires a bit of
@@ -115,15 +370,27 @@ You can do this by going through these steps:
 - Make a new PR with the change
 - Have the PR get reviewed and merged
 
-### Known installation issues
+## Known Installation Issues
+
+### Python Installation Errors
 
 On M1 Macs, if you get a `fatal error: 'Python.h' file not found` message, try a
-different method of installing Python. Installation via `pyenv` is known to
+different method of installing Python. The recommended approach is to use
-work.
+[`pyenv`](https://github.com/pyenv/pyenv), as noted above in the installation
+instructions.
+
+If you're using PyCharm for Python development, we've noticed some quirkiness
+with the IDE and the interaction between Poetry and virtual environment
+management that could cause a variety of problems to come up during project
+setup and dependency management. Other tools, such as Visual Studio Code, have
+proven to be a smoother experience for folks.
+
+### PostgreSQL Installation Errors
 
 A direct installation of PostgreSQL will not put the `createdb` command on your
 `$PATH`. It can be added there in your shell startup script, or a
-Homebrew-managed installation of PostgreSQL will take care of it.
+Homebrew-managed installation of PostgreSQL will take care of it. See the
+instructions above for more details.
 
 ## Documentation
 

sample.env

@@ -39,13 +39,14 @@ NOTIFY_APP_NAME=api
 
 #############################################################
 
+# E2E Testing
+
+NOTIFY_E2E_TEST_EMAIL=example@fake.gov
+NOTIFY_E2E_TEST_PASSWORD="don't write secrets to the sample file"
+
+#############################################################
+
 # Flask
 FLASK_APP=application.py
 FLASK_DEBUG=true
 WERKZEUG_DEBUG_PIN=off
-
-#############################################################
-
-# New Relic
-NEW_RELIC_CONFIG_FILE=newrelic.ini
-NEW_RELIC_LICENSE_KEY="don't write secrets to the sample file"