notifications-api/docs/infra-overview.md

Infrastructure overview

A diagram of the system is available in our compliance repo.

Notify is a Flask application running on cloud.gov, which also brokers access to a PostgreSQL database and Redis store.

In addition to the Flask app, Notify uses Celery to manage the task queue. Celery stores tasks in Redis.

GitHub Repositories

Application, infrastructure, and compliance work is spread across several repositories:

Application

• notifications-api for the API app
• notifications-admin for the Admin UI app
• notifications-utils for common library functions

Infrastructure

In addition to terraform directories in the api and admin apps above:

We maintain:

• usnotify-ssb A supplemental service broker that provisions SES and SNS for us
• ttsnotify-brokerpak-sms The brokerpak defining SNS (SMS sending)

We use:

• datagov-brokerpak-smtp The brokerpak defining SES
• cg-egress-proxy The caddy proxy that allows external API calls

Compliance

• us-notify-compliance for OSCAL control documentation and diagrams

Terraform

Development

There are several remote services required for local development:

• s3
• ses
• sns

Credentials for these services are created by running:

1. cd terraform/development
2. ./run.sh

in both the api repository as well as the admin repository.

This will append credentials to your .env file. You will need to manually clean up any prior runs from that file if you run that command again.

You can remove your development infrastructure by running ./run.sh -d

Resetting

./reset.sh can be used to import your development infrastructure information in case of a new computer or new working tree and the old terraform state file was not transferred.

Offboarding

./reset.sh -u USER_TO_OFFBOARD can be used to import another user's development resources in order to clean them up. Steps for use:

1. Move your existing terraform state file aside temporarily, so it is not overwritten.
2. ./reset.sh -u USER_TO_OFFBOARD
3. Answer no to the prompt about creating missing resources.
4. Run ./run.sh -u USER_TO_OFFBOARD -d to fully remove the rest of that user's resources.

Cloud.gov

The cloud.gov environment is configured with Terraform. See the terraform folder to learn about that.

AWS

In addition to services provisioned through cloud.gov, we have several services provisioned via supplemental service brokers in AWS. Our AWS services are currently located in several regions using Studio-controlled AWS accounts.

To send messages, we use Amazon Web Services SNS and SES. In addition, we use AWS Pinpoint to provision and manage phone numbers, short codes, and long codes for sending SMS.

In SNS, we have 3 topics for SMS receipts. These are not currently functional, so senders won't know the status of messages.

Through Pinpoint, the API needs at least one number so that the application itself can send SMS for authentication codes.

The API also has access to AWS S3 buckets for storing CSVs of messages and contact lists. It does not access a third S3 bucket that stores agency logos.

New Relic

We are using New Relic for application monitoring and error reporting. When requesting access to New Relic, ask to be added to the Benefits-Studio subaccount.

Onboarding

• Join the GSA GitHub org
• Get permissions for the repos
• Get access to the cloud.gov org && spaces
• Get access to AWS, if necessary
• Get access to New Relic, if necessary
• Create the local .env file by copying sample.env and running ./run.sh within the terraform/development folder
• Do stuff!

Setting up the infrastructure

These steps are required for new cloud.gov environments. Local development borrows SES & SNS infrastructure from the notify-staging cloud.gov space, so these steps are not required for new developers.

Steps to do a clean prod deploy to cloud.gov

Steps for deploying production from scratch. These can be updated for a new cloud.gov environment by subbing out prod or production for your desired environment within the steps.

1. Deploy API app
   1. Update terraform-production.yml and deploy-prod.yml to point to the correct space and git branch.
   2. Ensure that the domain module is commented out in terraform/production/main.tf
   3. Run CI/CD pipeline on the production branch by opening a PR from main to production
   4. Create any necessary DNS records (check notify-api-ses-production service credentials for instructions) within https://github.com/18f/dns
   5. Follow the Steps to prepare SES below
   6. (Optional) if using a public API route, uncomment the domain module and re-trigger a deploy
2. Deploy Admin app
   1. Update terraform-production.yml and deploy-prod.yml to point to the correct space and git branch.
   2. Ensure that the api_network_route and domain modules are commented out in terraform/production/main.tf
   3. Run CI/CD pipeline on the production branch by opening a PR from main to production
   4. Uncomment the api_network_route and domain modules and re-trigger a deploy
   5. Create DNS records for domain module within https://github.com/18f/dns

Steps to prepare SES

1. After the first deploy of the application with the SSB-brokered SES service completes:
   1. Log into the SES console and navigate to the SNS subscription page.
   2. Select "Request confirmation" for any subscriptions still in "Pending Confirmation" state
2. Find and replace instances in the repo of "testsender", "testreceiver" and "dispostable.com", with your origin and destination email addresses, which you verified in step 1 above.

TODO: create env vars for these origin and destination email addresses for the root service, and create new migrations to update postgres seed fixtures

Steps to prepare SNS

Move SNS out of sandbox.

This should be complete for all regions U.S. Notify has been deployed to or is currently planned to be deployed to.

1. Visit the SNS console for the region you will be sending from. Notes:
   1. SNS settings are per-region, so each environment must have its own region
   2. Pinpoint and SNS have confusing regional availability, so ensure both are available before submitting any requests.
2. Choose Text messaging (SMS) from the sidebar
3. Click the Exit SMS Sandbox button and submit the support request. This request should take at most a day to complete. Be sure to request a higher sending limit at the same time.

Request new phone numbers

1. Go to Pinpoint console for the same region you are using SNS in.
2. In the lefthand sidebar, go the SMS and Voice (bottom) and choose Phone Numbers
3. Under Number Settings choose Request Phone Number
4. Choose Toll-free number, tick SMS, untick Voice, choose transactional, hit next and then request
5. Select Toll-free registrations and Create registration
6. Select the number you just created and then Register existing toll-free number
7. Complete and submit the form. Approval usually takes about 2 weeks.
8. See the run book for information on how to set those numbers.

Example answers for toll-free registration form