Add basic terraform bootstrap and staging environments

.gitignore

@@ -81,3 +81,10 @@ varsfile*
 .secret*
 
 /scripts/run_my_tests.sh
+
+# Terraform
+.terraform.lock.hcl
+**/.terraform/*
+secrets.auto.tfvars
+terraform.tfstate
+terraform.tfstate.backup

terraform/README.md

@@ -0,0 +1,130 @@
+# Terraform
+
+This directory holds the terraform modules for maintaining your complete persistent infrastructure.
+
+Prerequisite: install the `jq` JSON processor: `brew install jq`
+
+## Initial setup
+
+1. Manually run the bootstrap module following instructions under `Terraform State Credentials`
+1. Setup CI/CD Pipeline to run Terraform
+  1. Copy bootstrap credentials to your CI/CD secrets using the instructions in the base README
+  1. Create a cloud.gov SpaceDeployer by following the instructions under `SpaceDeployers`
+  1. Copy SpaceDeployer credentials to your CI/CD secrets using the instructions in the base README
+1. Manually Running Terraform
+  1. Follow instructions under `Set up a new environment` to create your infrastructure
+
+## Terraform State Credentials
+
+The bootstrap module is used to create an s3 bucket for later terraform runs to store their state in.
+
+### Bootstrapping the state storage s3 buckets for the first time
+
+1. Run `terraform init`
+1. Run `./run.sh plan` to verify that the changes are what you expect
+1. Run `./run.sh apply` to set up the bucket and retrieve credentials
+1. Follow instructions under `Use bootstrap credentials`
+1. Ensure that `import.sh` includes a line and correct IDs for any resources created
+1. Run `./teardown_creds.sh` to remove the space deployer account used to create the s3 bucket
+
+### To make changes to the bootstrap module
+
+*This should not be necessary in most cases*
+
+1. Run `terraform init`
+1. If you don't have terraform state locally:
+    1. run `./import.sh`
+    1. optionally run `./run.sh apply` to include the existing outputs in the state file
+1. Make your changes
+1. Continue from step 2 of the boostrapping instructions
+
+### Retrieving existing bucket credentials
+
+1. Run `./run.sh show`
+1. Follow instructions under `Use bootstrap credentials`
+
+#### Use bootstrap credentials
+
+1. Add the following to `~/.aws/credentials`
+    ```
+    [notify-terraform-backend]
+    aws_access_key_id = <access_key_id from bucket_credentials>
+    aws_secret_access_key = <secret_access_key from bucket_credentials>
+    ```
+
+1. Copy `bucket` from `bucket_credentials` output to the backend block of `staging/providers.tf` and `production/providers.tf`
+
+## SpaceDeployers
+
+A [SpaceDeployer](https://cloud.gov/docs/services/cloud-gov-service-account/) account is required to run terraform or
+deploy the application from the CI/CD pipeline. Create a new account by running:
+
+`./create_service_account.sh -s <SPACE_NAME> -u <ACCOUNT_NAME>`
+
+## Set up a new environment manually
+
+The below steps rely on you first configuring access to the Terraform state in s3 as described in [Terraform State Credentials](#terraform-state-credentials).
+
+1. `cd` to the environment you are working in
+
+1. Set up a SpaceDeployer
+    ```bash
+    # create a space deployer service instance that can log in with just a username and password
+    # the value of < SPACE_NAME > should be `staging` or `prod` depending on where you are working
+    # the value for < ACCOUNT_NAME > can be anything, although we recommend
+    # something that communicates the purpose of the deployer
+    # for example: circleci-deployer for the credentials CircleCI uses to
+    # deploy the application or <your_name>-terraform for credentials to run terraform manually
+    ./create_service_account.sh -s <SPACE_NAME> -u <ACCOUNT_NAME> > secrets.auto.tfvars
+    ```
+
+    The script will output the `username` (as `cf_user`) and `password` (as `cf_password`) for your `<ACCOUNT_NAME>`. Read more in the [cloud.gov service account documentation](https://cloud.gov/docs/services/cloud-gov-service-account/).
+
+    The easiest way to use this script is to redirect the output directly to the `secrets.auto.tfvars` file it needs to be used in
+
+1. Run terraform from your new environment directory with
+    ```bash
+    terraform init
+    terraform plan
+    ```
+
+1. Apply changes with `terraform apply`.
+
+1. Remove the space deployer service instance if it doesn't need to be used again, such as when manually running terraform once.
+    ```bash
+    # <SPACE_NAME> and <ACCOUNT_NAME> have the same values as used above.
+    ./destroy_service_account.sh -s <SPACE_NAME> -u <ACCOUNT_NAME>
+    ```
+
+## Structure
+
+Each environment has its own module, which relies on a shared module for everything except the providers code and environment specific variables and settings.
+
+```
+- bootstrap/
+  |- main.tf
+  |- providers.tf
+  |- variables.tf
+  |- run.sh
+  |- teardown_creds.sh
+  |- import.sh
+- <env>/
+  |- main.tf
+  |- providers.tf
+  |- secrets.auto.tfvars
+  |- variables.tf
+```
+
+In the environment-specific modules:
+- `providers.tf` lists the required providers
+- `main.tf` calls the shared Terraform code, but this is also a place where you can add any other services, resources, etc, which you would like to set up for that environment
+- `variables.tf` lists the variables that will be needed, either to pass through to the child module or for use in this module
+- `secrets.auto.tfvars` is a file which contains the information about the service-key and other secrets that should not be shared
+
+In the bootstrap module:
+- `providers.tf` lists the required providers
+- `main.tf` sets up s3 bucket to be shared across all environments. It lives in `prod` to communicate that it should not be deleted
+- `variables.tf` lists the variables that will be needed. Most values are hard-coded in this module
+- `run.sh` Helper script to set up a space deployer and run terraform. The terraform action (`show`/`plan`/`apply`/`destroy`) is passed as an argument
+- `teardown_creds.sh` Helper script to remove the space deployer setup as part of `run.sh`
+- `import.sh` Helper script to create a new local state file in case terraform changes are needed

terraform/bootstrap/import.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+read -p "Are you sure you want to import terraform state (y/n)? " verify
+
+if [[ $verify == "y" ]]; then
+  echo "Importing bootstrap state"
+  ./run.sh import module.s3.cloudfoundry_service_instance.bucket 31204bcc-aae3-4cd3-8b59-5055a338d44f
+  ./run.sh import cloudfoundry_service_key.bucket_creds 483a6ac5-4ba0-48ad-9850-ef87b51aaa08
+  ./run.sh plan
+else
+  echo "Not importing bootstrap state"
+fi

terraform/bootstrap/main.tf

@@ -0,0 +1,24 @@
+locals {
+  cf_api_url      = "https://api.fr.cloud.gov"
+  s3_service_name = "notify-terraform-state"
+}
+
+module "s3" {
+  source = "github.com/18f/terraform-cloudgov//s3"
+
+  cf_api_url      = local.cf_api_url
+  cf_user         = var.cf_user
+  cf_password     = var.cf_password
+  cf_org_name     = "gsa-10x-prototyping"
+  cf_space_name   = "10x-notifications"
+  s3_service_name = local.s3_service_name
+}
+
+resource "cloudfoundry_service_key" "bucket_creds" {
+  name             = "${local.s3_service_name}-access"
+  service_instance = module.s3.bucket_id
+}
+
+output "bucket_credentials" {
+  value = cloudfoundry_service_key.bucket_creds.credentials
+}

terraform/bootstrap/providers.tf

@@ -0,0 +1,16 @@
+terraform {
+  required_version = "~> 1.0"
+  required_providers {
+    cloudfoundry = {
+      source  = "cloudfoundry-community/cloudfoundry"
+      version = "0.15.5"
+    }
+  }
+}
+
+provider "cloudfoundry" {
+  api_url      = local.cf_api_url
+  user         = var.cf_user
+  password     = var.cf_password
+  app_logs_max = 30
+}

terraform/bootstrap/run.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+if [[ ! -f "secrets.auto.tfvars" ]]; then
+  ../create_service_account.sh -s 10x-notifications -u config-bootstrap-deployer > secrets.auto.tfvars
+fi
+
+if [[ $# -gt 0 ]]; then
+  echo "Running terraform $@"
+  terraform $@
+else
+  echo "Not running terraform"
+fi

terraform/bootstrap/teardown_creds.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+../destroy_service_account.sh -s 10x-notifications -u config-bootstrap-deployer
+
+rm secrets.auto.tfvars

terraform/bootstrap/variables.tf

@@ -0,0 +1,2 @@
+variable "cf_password" {}
+variable "cf_user" {}

terraform/create_service_account.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+
+org="gsa-10x-prototyping"
+
+usage="
+$0: Create a Service User Account for a given space
+
+Usage:
+  $0 -h
+  $0 -s <SPACE NAME> -u <USER NAME> [-r <ROLE NAME>] [-o <ORG NAME>]
+
+Options:
+-h: show help and exit
+-s <SPACE NAME>: configure the space to act on. Required
+-u <USER NAME>: set the service user name. Required
+-r <ROLE NAME>: set the service user's role to either space-deployer or space-auditor. Default: space-deployer
+-o <ORG NAME>: configure the organization to act on. Default: $org
+"
+
+set -e
+set -o pipefail
+
+space=""
+service=""
+role="space-deployer"
+
+while getopts ":hs:u:r:o:" opt; do
+  case "$opt" in
+    s)
+      space=${OPTARG}
+      ;;
+    u)
+      service=${OPTARG}
+      ;;
+    r)
+      role=${OPTARG}
+      ;;
+    o)
+      org=${OPTARG}
+      ;;
+    h)
+      echo "$usage"
+      exit 0
+      ;;
+  esac
+done
+
+if [[ $space = "" || $service = "" ]]; then
+  echo "$usage"
+  exit 1
+fi
+
+cf target -o $org -s $space 1>&2
+
+# create user account service
+cf create-service cloud-gov-service-account $role $service 1>&2
+
+# create service key
+cf create-service-key $service service-account-key 1>&2
+
+# output service key to stdout in secrets.auto.tfvars format
+creds=`cf service-key $service service-account-key | tail -n 4`
+username=`echo $creds | jq '.username'`
+password=`echo $creds | jq '.password'`
+
+cat << EOF
+# generated with $0 -s $space -u $service -r $role -o $org
+# revoke with $(dirname $0)/destroy_service_account.sh -s $space -u $service -o $org
+
+cf_user = $username
+cf_password = $password
+EOF

terraform/destroy_service_account.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+
+org="gsa-10x-prototyping"
+
+usage="
+$0: Destroy a Service User Account in a given space
+
+Usage:
+  $0 -h
+  $0 -s <SPACE NAME> -u <USER NAME> [-o <ORG NAME>]
+
+Options:
+-h: show help and exit
+-s <SPACE NAME>: configure the space to act on. Required
+-u <USER NAME>: configure the service user name to destroy. Required
+-o <ORG NAME>: configure the organization to act on. Default: $org
+"
+
+set -e
+
+space=""
+service=""
+
+while getopts ":hs:u:o:" opt; do
+  case "$opt" in
+    s)
+      space=${OPTARG}
+      ;;
+    u)
+      service=${OPTARG}
+      ;;
+    o)
+      org=${OPTARG}
+      ;;
+    h)
+      echo "$usage"
+      exit 0
+      ;;
+  esac
+done
+
+if [[ $space = "" || $service = "" ]]; then
+  echo "$usage"
+  exit 1
+fi
+
+cf target -o $org -s $space
+
+# destroy service key
+cf delete-service-key $service service-account-key -f
+
+# destroy service
+cf delete-service $service -f

terraform/production/main.tf

@@ -0,0 +1,54 @@
+locals {
+  cf_org_name      = "TKTK"
+  cf_space_name    = "TKTK"
+  env              = "production"
+  app_name         = "notifications-api"
+  recursive_delete = false
+}
+
+module "database" {
+  source = "github.com/18f/terraform-cloudgov//database"
+
+  cf_user          = var.cf_user
+  cf_password      = var.cf_password
+  cf_org_name      = local.cf_org_name
+  cf_space_name    = local.cf_space_name
+  env              = local.env
+  app_name         = local.app_name
+  recursive_delete = local.recursive_delete
+  rds_plan_name    = "TKTK-production-rds-plan"
+}
+
+module "redis" {
+  source = "github.com/18f/terraform-cloudgov//redis"
+
+  cf_user          = var.cf_user
+  cf_password      = var.cf_password
+  cf_org_name      = local.cf_org_name
+  cf_space_name    = local.cf_space_name
+  env              = local.env
+  app_name         = local.app_name
+  recursive_delete = local.recursive_delete
+  redis_plan_name  = "TKTK-production-redis-plan"
+}
+
+###########################################################################
+# The following lines need to be commented out for the initial `terraform apply`
+# It can be re-enabled after:
+# 1) the app has first been deployed
+# 2) the route has been manually created by an OrgManager:
+#     `cf create-domain TKTK-org-name TKTK-production-domain-name`
+###########################################################################
+# module "domain" {
+#   source = "github.com/18f/terraform-cloudgov//domain"
+#
+#   cf_user          = var.cf_user
+#   cf_password      = var.cf_password
+#   cf_org_name      = local.cf_org_name
+#   cf_space_name    = local.cf_space_name
+#   env              = local.env
+#   app_name         = local.app_name
+#   recursive_delete = local.recursive_delete
+#   cdn_plan_name    = "domain"
+#   domain_name      = "TKTK-production-domain-name"
+# }

terraform/production/providers.tf

@@ -0,0 +1,17 @@
+terraform {
+  required_version = "~> 1.0"
+  required_providers {
+    cloudfoundry = {
+      source  = "cloudfoundry-community/cloudfoundry"
+      version = "0.15.5"
+    }
+  }
+
+  backend "s3" {
+    bucket  = "cg-31204bcc-aae3-4cd3-8b59-5055a338d44f"
+    key     = "api.tfstate.prod"
+    encrypt = "true"
+    region  = "us-gov-west-1"
+    profile = "notify-terraform-backend"
+  }
+}

terraform/production/variables.tf

@@ -0,0 +1,2 @@
+variable "cf_password" {}
+variable "cf_user" {}

terraform/set_space_egress.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+org="gsa-10x-prototyping"
+
+usage="
+$0: Set egress rules for given space
+
+Usage:
+  $0 -h
+  $0 -s <SPACE NAME> [-o <ORG NAME>] [-p] [-t]
+
+Options:
+-h: show help and exit
+-s <SPACE NAME>: configure the space to act on. Required
+-o <ORG NAME>: configure the organization to act on. Default: $org
+-p: Add the public egress rules
+-t: Add the trusted egress rules
+
+Notes:
+* If -p or -t are not passed, the related security groups will be removed, if they were present
+"
+
+set -e
+
+space=""
+public=false
+trusted=false
+
+while getopts ":hs:o:pt" opt; do
+  case "$opt" in
+    s)
+      space=${OPTARG}
+      ;;
+    o)
+      org=${OPTARG}
+      ;;
+    p)
+      public=true
+      ;;
+    t)
+      trusted=true
+      ;;
+    h)
+      echo "$usage"
+      exit 0
+      ;;
+  esac
+done
+
+if [[ $space = "" ]]; then
+  echo "$usage"
+  exit 1
+fi
+
+if [[ $public = true ]]; then
+  cf bind-security-group public_networks_egress $org --space $space
+else
+  cf unbind-security-group public_networks_egress $org $space
+fi
+
+if [[ $trusted = true ]]; then
+  cf bind-security-group trusted_local_networks_egress $org --space $space
+else
+  cf unbind-security-group trusted_local_networks_egress $org $space
+fi
+
+echo "Done"

terraform/staging/main.tf

@@ -0,0 +1,33 @@
+locals {
+  cf_org_name      = "gsa-10x-prototyping"
+  cf_space_name    = "10x-notifications"
+  env              = "staging"
+  app_name         = "notifications-api"
+  recursive_delete = true
+}
+
+module "database" {
+  source = "github.com/18f/terraform-cloudgov//database"
+
+  cf_user          = var.cf_user
+  cf_password      = var.cf_password
+  cf_org_name      = local.cf_org_name
+  cf_space_name    = local.cf_space_name
+  env              = local.env
+  app_name         = local.app_name
+  recursive_delete = local.recursive_delete
+  rds_plan_name    = "micro-psql"
+}
+
+module "redis" {
+  source = "github.com/18f/terraform-cloudgov//redis"
+
+  cf_user          = var.cf_user
+  cf_password      = var.cf_password
+  cf_org_name      = local.cf_org_name
+  cf_space_name    = local.cf_space_name
+  env              = local.env
+  app_name         = local.app_name
+  recursive_delete = local.recursive_delete
+  redis_plan_name  = "redis-dev"
+}

terraform/staging/providers.tf

@@ -0,0 +1,17 @@
+terraform {
+  required_version = "~> 1.0"
+  required_providers {
+    cloudfoundry = {
+      source  = "cloudfoundry-community/cloudfoundry"
+      version = "0.15.5"
+    }
+  }
+
+  backend "s3" {
+    bucket  = "cg-31204bcc-aae3-4cd3-8b59-5055a338d44f"
+    key     = "api.tfstate.stage"
+    encrypt = "true"
+    region  = "us-gov-west-1"
+    profile = "notify-terraform-backend"
+  }
+}

terraform/staging/variables.tf

@@ -0,0 +1,2 @@
+variable "cf_password" {}
+variable "cf_user" {}