bump owasp job version

Bumps [dnspython](https://github.com/rthalley/dnspython) from 2.2.1 to 2.3.0.
- [Changelog](https://github.com/rthalley/dnspython/blob/master/doc/whatsnew.rst)
- [Commits](https://github.com/rthalley/dnspython/compare/v2.2.1...v2.3.0)

---
updated-dependencies:
- dependency-name: dnspython
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

.cfignore

@@ -1 +0,0 @@
-.gitignore

.cfignore

@@ -0,0 +1,109 @@
+# from deploy-exclude.lst
+
+*__pycache__*
+.git/*
+app/assets/*
+bower_components/*
+cache/*
+.cache/*
+node_modules/*
+target/*
+venv/*
+build/*
+.envrc
+tests/.cache/*
+.cf/*
+
+# from .gitignore
+
+queues.csv
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+.venv/
+venv/
+venv-freeze/
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+/cache
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+.pytest_cache
+coverage.xml
+test_results.xml
+*,cover
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+.idea/
+.vscode
+
+# Mac
+*.DS_Store
+environment.sh
+.envrc
+.env
+.env*
+varsfile
+
+celerybeat-schedule
+
+# CloudFoundry
+.cf
+varsfile*
+.secret*
+
+/scripts/run_my_tests.sh
+
+# Terraform
+.terraform.lock.hcl
+**/.terraform/*
+secrets.auto.tfvars
+terraform.tfstate
+terraform.tfstate.backup

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/create-a-new-adr-from-template.md

@@ -0,0 +1,125 @@
+---
+name: Create a new ADR from template
+about: Use this template to create a new Architectural Decision Record.
+title: ADR
+labels: documentation
+assignees: ''
+
+---
+
+# TITLE:  ADR Title Here
+
+
+| CREATED DATE | LAST UPDATED | STATUS | IMPLEMENTED | AUTHOR | STAKEHOLDERS |
+| :---: | :---: | :---: | :---: | :---: |
+| Date when ADR was created - MM/DD/YYYY format | Date when ADR was last updated - MM/DD/YYYY format - or N/A | Current ADR status - one of Proposed /  Accepted / Rejected / Deprecated / Superseded By ADR - link to ADR | Yes or No | GitHub username(s) of author(s) | GitHub username(s) or team name(s) of other folks involved |
+
+
+## CONTEXT AND PROBLEM STATEMENT
+
+**OPEN ISSUE(S):** Link(s) to related issue(s) here, if applicable.
+
+Explain the nature of the problem, challenge, or decision to be made here. Be as
+succinct as possible, but don't leave out any details.  State things matter of
+factly and as objectively as you can.
+
+
+## DECISION DRIVERS
+
+List anything that plays a major role in making a decision here.  These could
+be one or more of the following:
+
+- Desired qualities
+- Desired outcomes
+- Primary concerns
+- Constraints
+- ... (anything else pertinent to the decision-making process)
+
+
+### SECURITY COMPLIANCE CONSIDERATIONS
+
+Because we work in a regulated space with many compliance requirements, we need
+to make sure we're accounting for any security concerns and adhering to all
+security compliance requirements.  List them in this section along with any
+relevant details:
+
+- Security concern 1
+  - Concern detail 1
+  - Concern detail 2
+  - Concern detail ...
+
+- Security concern 2
+  - Concern detail 1
+  - Concern detail 2
+  - Concern detail ...
+
+
+## CONSIDERED OPTIONS
+
+List all options that have either been discussed or thought of as a potential
+solution to the context and problem statement.  Include any pros and cons with
+each option, like so:
+
+- **Name of first option:**  A brief summary of the option.
+  - Pros:
+    - Pro 1
+    - Pro 2
+    - Pro ...
+
+  - Cons:
+    - Con 1
+    - Con 2
+    - Con ...
+
+- **Name of second option:**  A brief summary of the option.
+  - Pros:
+    - Pro 1
+    - Pro 2
+    - Pro ...
+
+  - Cons:
+    - Con 1
+    - Con 2
+    - Con ...
+
+
+## PROPOSED OR CHOSEN OPTION:  Proposed/Chosen Option Title Here
+
+Summarize the decision for the proposed/chosen option here.  Be as concise and
+objective as possible while including all relevant details so that a clear
+justification is provided.  Include a list of consequences for choosing this
+option, both positive and negative:
+
+
+### Consequences
+
+- Positive
+  - Positive consequence 1
+  - Positive consequence 2
+  - Positive consequence ...
+
+- Negative
+  - Negative consequence 1
+  - Negative consequence 2
+  - Negative consequence ...
+
+
+## VALIDATION AND NEXT STEPS
+
+This section likely won't be filled out until a decision has been made by the
+team and the ADR is accepted.  If this comes to pass, then write up the criteria
+that would ensure this ADR is both implemented and tested correctly and
+adequately.  This could be a short summary and/or a list of things:
+
+- **Criterion name 1:**  Description of criterion 1
+  - Requirement or action 1
+  - Requirement or action 2
+  - Requirement or action ...
+
+- **Criterion name 2:**  Description of criterion 2
+  - Requirement or action 1
+  - Requirement or action 2
+  - Requirement or action ...
+
+Lastly, include a link(s) to an issue(s) that represent the work that will
+take place as follow-ups to this ADR.

.github/ISSUE_TEMPLATE/create-new-adr-form.yml

@@ -0,0 +1,130 @@
+name: Create a new ADR
+description: Create a new Architectural Decision Record.
+title: "[ADR]: "
+labels: ["documentation"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Use this form to create a new Architectural Decision Record.
+  - type: input
+    id: title
+    attributes:
+      label: ADR Title
+      description: What is the title of this ADR?
+      placeholder: e.g., Establishing ADRs for US Notify
+    validations:
+      required: true
+  - type: input
+    id: created_date
+    attributes:
+      label: Created Date
+      description: What is the date this ADR is being created?
+      placeholder: MM/DD/YYYY
+    validations:
+      required: true
+  - type: dropdown
+    id: status
+    attributes:
+      label: Status
+      description: What is the current status of this ADR?
+      options:
+        - Proposed
+        - Accepted
+        - Rejected
+        - Deprecated
+        - Superseded By
+    validations:
+      required: true
+  - type: dropdown
+    id: implemented
+    attributes:
+      label: Implemented
+      description: Is this ADR implemented?
+      options:
+        - 'Yes'
+        - 'No'
+    validations:
+      required: true
+  - type: input
+    id: superseded_by
+    attributes:
+      label: Superseded By
+      description: If this ADR is superseded by another, which one is it?
+      placeholder: e.g., ADR-0001
+    validations:
+      required: false
+  - type: input
+    id: author
+    attributes:
+      label: Author
+      description: Who is the author(s) of this ADR?
+      placeholder: Enter your GitHub username here, and anyone else helping author this
+    validations:
+      required: true
+  - type: input
+    id: stakeholders
+    attributes:
+      label: Stakeholders
+      description: Who are the stakeholders of this ADR?
+      placeholder: Enter their GitHub username(s) here
+    validations:
+      required: false
+  - type: input
+    id: linked-github-issue
+    attributes:
+      label: Related Issue(s)
+      description: List any related GitHub issues to this ADR here (issue number only).
+      placeholder: e.g., 288
+    validations:
+      required: false
+  - type: textarea
+    id: context-and-problem-statement
+    attributes:
+      label: Context and Problem Statement
+      description: Explain the nature of the problem, challenge, or decision to be made here.
+      placeholder: Be as succinct as possible, but do not leave out any details.  State things matter of factly and as objectively as you can.
+    validations:
+      required: true
+  - type: textarea
+    id: decision-drivers
+    attributes:
+      label: Decision Drivers
+      description: List anything that plays a major role in making a decision here.
+    validations:
+      required: true
+  - type: textarea
+    id: security-compliance-considerations
+    attributes:
+      label: Security Compliance Considerations
+      description: List any security concerns in this section along with any relevant details.
+    validations:
+      required: true
+  - type: textarea
+    id: considered-options
+    attributes:
+      label: Considered Options
+      description: List all options that have either been discussed or thought of as a potential solution to the context and problem statement.  Include any pros and cons with each option.
+    validations:
+      required: true
+  - type: input
+    id: proposed-option
+    attributes:
+      label: Proposed Option
+      description: If an option is already clear or has been chosen, please name it here.
+    validations:
+      required: false
+  - type: textarea
+    id: proposed-option-justification
+    attributes:
+      label: Proposed or Chosen Option
+      description: Summarize the decision for the proposed/chosen option here, including a list of positive and negative consequences.
+    validations:
+      required: false
+  - type: textarea
+    id: validation-and-next-steps
+    attributes:
+      label: Validation and Next Steps
+      description: If an option has already been chosen and justified, summarize or list the validation and next steps that came from the decision.
+    validations:
+      required: false

.github/actions/deploy-proxy/action.yml

@@ -0,0 +1,42 @@
+name: Deploy egress proxy
+description: Set egress space security groups and deploy proxy
+inputs:
+  cf_space:
+    description: The space the target app exists in.
+    required: true
+  app:
+    description: application name to be proxied.
+    required: true
+  proxy_repo:
+    description: git repo for cg-egress-proxy
+    default: https://github.com/GSA-TTS/cg-egress-proxy.git
+  proxy_version:
+    description: git ref to be deployed
+    default: main
+runs:
+  using: composite
+  steps:
+    - name: Set restricted space egress
+      shell: bash
+      run: ./terraform/set_space_egress.sh -t -s ${{ inputs.cf_space }}
+    - name: Set public space egress
+      shell: bash
+      run: ./terraform/set_space_egress.sh -p -s ${{ inputs.cf_space }}-egress
+    - name: Create temp directory
+      shell: bash
+      id: create-temp-dir
+      run: echo "path=$(mktemp -d -t egress-XXXXXXXXXX --tmpdir=$RUNNER_TEMP)" >> $GITHUB_OUTPUT
+    - name: Clone cg-egress-proxy
+      shell: bash
+      run: git clone ${{ inputs.proxy_repo }} ${{ steps.create-temp-dir.outputs.path }}
+    - name: Switch to deploy ref
+      shell: bash
+      working-directory: ${{ steps.create-temp-dir.outputs.path }}
+      run: git checkout ${{ inputs.proxy_version }}
+    - name: Copy config files
+      shell: bash
+      run: cp ./deploy-config/egress_proxy/${{ inputs.app }}.*.acl ${{ steps.create-temp-dir.outputs.path }}
+    - name: Deploy proxy
+      shell: bash
+      working-directory: ${{ steps.create-temp-dir.outputs.path }}
+      run: ./bin/cf-deployproxy -a ${{ inputs.app }} -p egress-proxy -e egress_proxy

.github/actions/setup-project/action.yml

@@ -0,0 +1,18 @@
+name: Set up project
+description: Setup python & install dependencies
+runs:
+  using: composite
+  steps:
+    - name: Install container dependencies
+      shell: bash
+      run: |
+        sudo apt-get update \
+        && sudo apt-get install -y --no-install-recommends \
+        libcurl4-openssl-dev
+    - name: Set up Python 3.9
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.9"
+    - name: Install pipenv
+      shell: bash
+      run: pip install --upgrade pipenv

.github/dependabot.yml

@@ -0,0 +1,12 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "pip" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "daily"
+

.github/workflows/checks.yml

@@ -0,0 +1,128 @@
+name: Run checks
+
+on: [push]
+
+permissions:
+  contents: read
+
+env:
+  DEBUG: True
+  NOTIFY_ENVIRONMENT: test
+  NEW_RELIC_CONFIG_FILE: newrelic.ini
+  NEW_RELIC_ENVIRONMENT: test
+  FLASK_APP: application.py
+  WERKZEUG_DEBUG_PIN: off
+  REDIS_ENABLED: 0
+  AWS_US_TOLL_FREE_NUMBER: "+18556438890"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_USER: user
+          POSTGRES_PASSWORD: password
+          POSTGRES_DB: test_notification_api
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          # Maps tcp port 5432 on service container to the host
+          - 5432:5432
+
+    steps:
+    - uses: actions/checkout@v3
+    - uses: ./.github/actions/setup-project
+    - name: Install application dependencies
+      run: make bootstrap
+      env:
+        SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+    - name: Run style checks
+      run: pipenv run flake8 .
+    - name: Check imports alphabetized
+      run: pipenv run isort --check-only ./app ./tests
+    - name: Run tests with coverage
+      run: pipenv run coverage run --omit=*/notifications_utils/* -m pytest -n4 --maxfail=10
+      env:
+        SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+    - name: Check coverage threshold
+      run: pipenv run coverage report --fail-under=50
+
+  validate-new-relic-config:
+    runs-on: ubuntu-latest
+    environment: staging
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Install pipenv packages
+        run: pipenv install --dev
+      - name: Validate NewRelic config
+        env:
+          NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
+          # Need to set a NEW_RELIC_ENVIRONMENT with monitor_mode: true
+          NEW_RELIC_ENVIRONMENT: staging
+        run: pipenv run newrelic-admin validate-config $NEW_RELIC_CONFIG_FILE
+
+  pip-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Create requirements.txt
+        run: pipenv requirements > requirements.txt
+      - uses: pypa/gh-action-pip-audit@v1.0.6
+        with:
+          inputs: requirements.txt
+
+  static-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Install bandit
+        run: pip install bandit
+      - name: Run scan
+        run: bandit -r app/ --confidence-level medium
+
+  dynamic-scan:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_USER: user
+          POSTGRES_PASSWORD: password
+          POSTGRES_DB: test_notification_api
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          # Maps tcp port 5432 on service container to the host
+          - 5432:5432
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Install application dependencies
+        run: make bootstrap
+        env:
+          SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+      - name: Run server
+        run: make run-flask &
+        env:
+          SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+      - name: Run OWASP Baseline Scan
+        uses: zaproxy/action-api-scan@v0.3.1
+        with:
+          docker_name: 'owasp/zap2docker-stable'
+          target: 'http://localhost:6011/docs/openapi.yml'
+          fail_action: true
+          allow_issue_writing: false
+          rules_file_name: 'zap.conf'
+          cmd_options: '-I'

.github/workflows/daily_checks.yml

@@ -0,0 +1,90 @@
+name: Run daily scans
+
+on:
+  schedule:
+    # cron format: 'minute hour dayofmonth month dayofweek'
+    # this will run at noon UTC every day (7am EST / 8am EDT)
+    - cron: '0 12 * * *'
+
+permissions:
+  contents: read
+
+env:
+  DEBUG: True
+  NOTIFY_ENVIRONMENT: test
+  NEW_RELIC_CONFIG_FILE: newrelic.ini
+  NEW_RELIC_ENVIRONMENT: test
+  FLASK_APP: application.py
+  WERKZEUG_DEBUG_PIN: off
+  REDIS_ENABLED: 0
+  AWS_US_TOLL_FREE_NUMBER: "+18556438890"
+
+jobs:
+  pip-audit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Create requirements.txt
+        run: pipenv requirements > requirements.txt
+      - uses: pypa/gh-action-pip-audit@v1.0.6
+        with:
+          inputs: requirements.txt
+      - name: Upload pip-audit artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: pip-audit-report
+          path: /tmp/pip-audit-output.txt
+
+  static-scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Install bandit
+        run: pip install bandit
+      - name: Run scan
+        run: bandit -r app/ -f txt -o /tmp/bandit-output.txt --confidence-level medium
+      - name: Upload bandit artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: bandit-report
+          path: /tmp/bandit-output.txt
+
+  dynamic-scan:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres
+        env:
+          POSTGRES_USER: user
+          POSTGRES_PASSWORD: password
+          POSTGRES_DB: test_notification_api
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          # Maps tcp port 5432 on service container to the host
+          - 5432:5432
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-project
+      - name: Install application dependencies
+        run: make bootstrap
+        env:
+          SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+      - name: Run server
+        run: make run-flask &
+        env:
+          SQLALCHEMY_DATABASE_TEST_URI: postgresql://user:password@localhost:5432/test_notification_api
+      - name: Run OWASP Baseline Scan
+        uses: zaproxy/action-api-scan@v0.1.1
+        with:
+          docker_name: 'owasp/zap2docker-weekly'
+          target: 'http://localhost:6011/docs/openapi.yml'
+          fail_action: true
+          allow_issue_writing: false
+          rules_file_name: 'zap.conf'
+          cmd_options: '-I'

.github/workflows/deploy-demo.yml

@@ -0,0 +1,83 @@
+name: Deploy to demo environment
+
+on:
+  push:
+    branches: [ production ]
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: demo
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+
+    - name: Check for changes to Terraform
+      id: changed-terraform-files
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          terraform/demo
+          terraform/shared
+          .github/workflows/deploy-demo.yml
+    - name: Terraform init
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/demo
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+      run: terraform init
+    - name: Terraform apply
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/demo
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+        TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+      run: terraform apply -auto-approve -input=false
+
+    - uses: ./.github/actions/setup-project
+    - name: Install application dependencies
+      run: make bootstrap
+
+    - name: Create requirements.txt because Cloud Foundry does a weird pipenv thing
+      run: pipenv requirements > requirements.txt
+
+    - name: Deploy to cloud.gov
+      uses: 18f/cg-deploy-action@main
+      env:
+        DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
+        SECRET_KEY: ${{ secrets.SECRET_KEY }}
+        ADMIN_CLIENT_SECRET: ${{ secrets.ADMIN_CLIENT_SECRET }}
+        NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
+      with:
+        cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
+        cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        cf_org: gsa-tts-benefits-studio-prototyping
+        cf_space: notify-demo
+        push_arguments: >-
+          --vars-file deploy-config/demo.yml
+          --var DANGEROUS_SALT="$DANGEROUS_SALT"
+          --var SECRET_KEY="$SECRET_KEY"
+          --var ADMIN_CLIENT_SECRET="$ADMIN_CLIENT_SECRET"
+          --var NEW_RELIC_LICENSE_KEY="$NEW_RELIC_LICENSE_KEY"
+
+    - name: Check for changes to egress config
+      id: changed-egress-config
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          deploy-config/egress_proxy/notify-api-demo.*.acl
+          .github/actions/deploy-proxy/action.yml
+          .github/workflows/deploy-demo.yml
+    - name: Deploy egress proxy
+      if: steps.changed-egress-config.outputs.any_changed == 'true'
+      uses: ./.github/actions/deploy-proxy
+      with:
+        cf_space: notify-demo
+        app: notify-api-demo

.github/workflows/deploy-prod.yml

@@ -0,0 +1,87 @@
+name: Deploy to production environment
+
+on:
+  push:
+    branches: [ production ]
+
+permissions:
+  contents: read
+
+# deploy-prod and deploy-demo will run in parallel now.
+# TODO: Research if we want to serialize them
+# by moving the jobs into a single file similar to
+# https://github.com/GSA/usnotify-ssb/blob/main/.github/workflows/apply.yml
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: production
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+
+    - name: Check for changes to Terraform
+      id: changed-terraform-files
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          terraform/production
+          terraform/shared
+          .github/workflows/deploy-prod.yml
+    - name: Terraform init
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/production
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+      run: terraform init
+    - name: Terraform apply
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/production
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+        TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+      run: terraform apply -auto-approve -input=false
+
+    - uses: ./.github/actions/setup-project
+    - name: Install application dependencies
+      run: make bootstrap
+
+    - name: Create requirements.txt because Cloud Foundry does a weird pipenv thing
+      run: pipenv requirements > requirements.txt
+
+    - name: Deploy to cloud.gov
+      uses: 18f/cg-deploy-action@main
+      env:
+        DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
+        SECRET_KEY: ${{ secrets.SECRET_KEY }}
+        ADMIN_CLIENT_SECRET: ${{ secrets.ADMIN_CLIENT_SECRET }}
+        NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
+      with:
+        cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
+        cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        cf_org: gsa-tts-benefits-studio-prototyping
+        cf_space: notify-production
+        push_arguments: >-
+          --vars-file deploy-config/production.yml
+          --var DANGEROUS_SALT="$DANGEROUS_SALT"
+          --var SECRET_KEY="$SECRET_KEY"
+          --var ADMIN_CLIENT_SECRET="$ADMIN_CLIENT_SECRET"
+          --var NEW_RELIC_LICENSE_KEY="$NEW_RELIC_LICENSE_KEY"
+
+    - name: Check for changes to egress config
+      id: changed-egress-config
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          deploy-config/egress_proxy/notify-api-production.*.acl
+          .github/actions/deploy-proxy/action.yml
+          .github/workflows/deploy-prod.yml
+    - name: Deploy egress proxy
+      if: steps.changed-egress-config.outputs.any_changed == 'true'
+      uses: ./.github/actions/deploy-proxy
+      with:
+        cf_space: notify-production
+        app: notify-api-production

.github/workflows/deploy.yml

@@ -0,0 +1,96 @@
+name: Deploy to staging environment
+
+on:
+  workflow_run:
+    workflows: [ Run checks ]
+    types:
+      - completed
+    branches: [ main ] # Redundant, workflow_run events are only triggered on default branch (`main`)
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    if: ${{ github.event.workflow_run.conclusion == 'success' }}
+
+    environment: staging
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+
+    - name: Check for changes to Terraform
+      id: changed-terraform-files
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          terraform/staging
+          terraform/shared
+          .github/workflows/deploy.yml
+    - name: Terraform init
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/staging
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+      run: terraform init
+    - name: Terraform apply
+      if: steps.changed-terraform-files.outputs.any_changed == 'true'
+      working-directory: terraform/staging
+      env:
+        AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+        TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+      run: terraform apply -auto-approve -input=false
+
+    - uses: ./.github/actions/setup-project
+    - name: Install application dependencies
+      run: make bootstrap
+
+    - name: Create requirements.txt because Cloud Foundry does a weird pipenv thing
+      run: pipenv requirements > requirements.txt
+
+    - name: Deploy to cloud.gov
+      uses: 18f/cg-deploy-action@main
+      env:
+        DANGEROUS_SALT: ${{ secrets.DANGEROUS_SALT }}
+        SECRET_KEY: ${{ secrets.SECRET_KEY }}
+        ADMIN_CLIENT_SECRET: ${{ secrets.ADMIN_CLIENT_SECRET }}
+        NEW_RELIC_LICENSE_KEY: ${{ secrets.NEW_RELIC_LICENSE_KEY }}
+      with:
+        cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
+        cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        cf_org: gsa-tts-benefits-studio-prototyping
+        cf_space: notify-staging
+        push_arguments: >-
+          --vars-file deploy-config/staging.yml
+          --var DANGEROUS_SALT="$DANGEROUS_SALT"
+          --var SECRET_KEY="$SECRET_KEY"
+          --var ADMIN_CLIENT_SECRET="$ADMIN_CLIENT_SECRET"
+          --var NEW_RELIC_LICENSE_KEY="$NEW_RELIC_LICENSE_KEY"
+
+    - name: Check for changes to egress config
+      id: changed-egress-config
+      uses: tj-actions/changed-files@v34
+      with:
+        files: |
+          deploy-config/egress_proxy/notify-api-staging.*.acl
+          .github/actions/deploy-proxy/action.yml
+          .github/workflows/deploy.yml
+    - name: Deploy egress proxy
+      if: steps.changed-egress-config.outputs.any_changed == 'true'
+      uses: ./.github/actions/deploy-proxy
+      with:
+        cf_space: notify-staging
+        app: notify-api-staging
+
+  bail:
+    runs-on: ubuntu-latest
+    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
+    steps:
+      - uses: actions/github-script@v6
+        with:
+          script: core.setFailed('Checks failed, not deploying')

.github/workflows/drift.yml

@@ -0,0 +1,66 @@
+name: Verify Infrastructure
+
+on:
+  schedule:
+    # cron format: 'minute hour dayofmonth month dayofweek'
+    # this will run at noon UTC every day (7am EST / 8am EDT)
+    - cron: '0 12 * * *'
+
+jobs:
+  check_staging_drift:
+    runs-on: ubuntu-latest
+    name: Check for drift of staging terraform configuration
+    environment: staging
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Check for drift
+        uses: dflook/terraform-check@v1
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        with:
+          path: terraform/staging
+
+  check_demo_drift:
+    runs-on: ubuntu-latest
+    name: Check for drift of demo terraform configuration
+    environment: demo
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: 'production'
+
+      - name: Check for drift
+        uses: dflook/terraform-check@v1
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        with:
+          path: terraform/demo
+
+  check_prod_drift:
+    runs-on: ubuntu-latest
+    name: Check for drift of production terraform configuration
+    environment: production
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: 'production'
+
+      - name: Check for drift
+        uses: dflook/terraform-check@v1
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        with:
+          path: terraform/production

.github/workflows/restage-apps.yml

@@ -0,0 +1,36 @@
+---
+name: Restage apps
+
+on:
+  workflow_dispatch:
+    inputs:
+      environment:
+        description: "Which environment needs to be restaged"
+        required: true
+        default: staging
+        type: environment
+
+jobs:
+  restage_apps:
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.environment }}
+    strategy:
+      matrix:
+        app: ["api", "admin"]
+    steps:
+      - name: Restage ${{matrix.app}}
+        uses: 18f/cg-deploy-action@main
+        with:
+          cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
+          cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+          cf_org: gsa-tts-benefits-studio-prototyping
+          cf_space: notify-${{ inputs.environment }}
+          full_command: "cf restage --strategy rolling notify-${{matrix.app}}-${{inputs.environment}}"
+      - name: Restage ${{matrix.app}} egress
+        uses: 18f/cg-deploy-action@main
+        with:
+          cf_username: ${{ secrets.CLOUDGOV_USERNAME }}
+          cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+          cf_org: gsa-tts-benefits-studio-prototyping
+          cf_space: notify-${{ inputs.environment }}-egress
+          full_command: "cf restage --strategy rolling egress-proxy-notify-${{matrix.app}}-${{inputs.environment}}"

.github/workflows/terraform-demo.yml

@@ -0,0 +1,79 @@
+name: Run Terraform plan in demo
+
+on:
+  pull_request:
+    branches: [ production ]
+    paths: [ 'terraform/**' ]
+
+defaults:
+  run:
+    working-directory: terraform/demo
+
+jobs:
+  terraform:
+    name: Terraform plan
+    runs-on: ubuntu-latest
+    environment: demo
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Terraform format
+        id: format
+        run: terraform fmt -check
+
+      - name: Terraform init
+        id: init
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        run: terraform init
+
+      - name: Terraform validate
+        id: validation
+        run: terraform validate -no-color
+
+      - name: Terraform plan
+        id: plan
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        run: terraform plan -no-color -input=false 2>&1 | tee plan_output.txt
+
+      - name: Read Terraform plan output file
+        id: terraform_output
+        uses: juliangruber/read-file-action@v1
+        if: ${{ always() }}
+        with:
+          path: ./terraform/demo/plan_output.txt
+
+      # inspiration: https://learn.hashicorp.com/tutorials/terraform/github-actions#review-actions-workflow
+      - name: Update PR
+        uses: actions/github-script@v6
+        # we would like to update the PR even when a prior step failed
+        if: ${{ always() }}
+        with:
+          script: |
+            const output = `Terraform Format and Style: ${{ steps.format.outcome }}
+            Terraform Initialization: ${{ steps.init.outcome }}
+            Terraform Validation: ${{ steps.validation.outcome }}
+            Terraform Plan: ${{ steps.plan.outcome }}
+
+            <details><summary>Show Plan</summary>
+
+            \`\`\`\n
+            ${{ steps.terraform_output.outputs.content }}
+            \`\`\`
+
+            </details>
+
+            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
+
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: output
+            })

.github/workflows/terraform-production.yml

@@ -0,0 +1,79 @@
+name: Run Terraform plan in production
+
+on:
+  pull_request:
+    branches: [ production ]
+    paths: [ 'terraform/**' ]
+
+defaults:
+  run:
+    working-directory: terraform/production
+
+jobs:
+  terraform:
+    name: Terraform plan
+    runs-on: ubuntu-latest
+    environment: production
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Terraform format
+        id: format
+        run: terraform fmt -check
+
+      - name: Terraform init
+        id: init
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        run: terraform init
+
+      - name: Terraform validate
+        id: validation
+        run: terraform validate -no-color
+
+      - name: Terraform plan
+        id: plan
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        run: terraform plan -no-color -input=false 2>&1 | tee plan_output.txt
+
+      - name: Read Terraform plan output file
+        id: terraform_output
+        uses: juliangruber/read-file-action@v1
+        if: ${{ always() }}
+        with:
+          path: ./terraform/production/plan_output.txt
+
+      # inspiration: https://learn.hashicorp.com/tutorials/terraform/github-actions#review-actions-workflow
+      - name: Update PR
+        uses: actions/github-script@v6
+        # we would like to update the PR even when a prior step failed
+        if: ${{ always() }}
+        with:
+          script: |
+            const output = `Terraform Format and Style: ${{ steps.format.outcome }}
+            Terraform Initialization: ${{ steps.init.outcome }}
+            Terraform Validation: ${{ steps.validation.outcome }}
+            Terraform Plan: ${{ steps.plan.outcome }}
+
+            <details><summary>Show Plan</summary>
+
+            \`\`\`\n
+            ${{ steps.terraform_output.outputs.content }}
+            \`\`\`
+
+            </details>
+
+            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
+
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: output
+            })

.github/workflows/terraform-staging.yml

@@ -0,0 +1,79 @@
+name: Run Terraform plan in staging
+
+on:
+  pull_request:
+    branches: [ main ]
+    paths: [ 'terraform/**' ]
+
+defaults:
+  run:
+    working-directory: terraform/staging
+
+jobs:
+  terraform:
+    name: Terraform plan
+    runs-on: ubuntu-latest
+    environment: staging
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Terraform format
+        id: format
+        run: terraform fmt -check
+
+      - name: Terraform init
+        id: init
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+        run: terraform init
+
+      - name: Terraform validate
+        id: validation
+        run: terraform validate -no-color
+
+      - name: Terraform plan
+        id: plan
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.TERRAFORM_STATE_ACCESS_KEY }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.TERRAFORM_STATE_SECRET_ACCESS_KEY }}
+          TF_VAR_cf_user: ${{ secrets.CLOUDGOV_USERNAME }}
+          TF_VAR_cf_password: ${{ secrets.CLOUDGOV_PASSWORD }}
+        run: terraform plan -no-color -input=false 2>&1 | tee plan_output.txt
+
+      - name: Read Terraform plan output file
+        id: terraform_output
+        uses: juliangruber/read-file-action@v1
+        if: ${{ always() }}
+        with:
+          path: ./terraform/staging/plan_output.txt
+
+      # inspiration: https://learn.hashicorp.com/tutorials/terraform/github-actions#review-actions-workflow
+      - name: Update PR
+        uses: actions/github-script@v6
+        # we would like to update the PR even when a prior step failed
+        if: ${{ always() }}
+        with:
+          script: |
+            const output = `Terraform Format and Style: ${{ steps.format.outcome }}
+            Terraform Initialization: ${{ steps.init.outcome }}
+            Terraform Validation: ${{ steps.validation.outcome }}
+            Terraform Plan: ${{ steps.plan.outcome }}
+
+            <details><summary>Show Plan</summary>
+
+            \`\`\`\n
+            ${{ steps.terraform_output.outputs.content }}
+            \`\`\`
+
+            </details>
+
+            *Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
+
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: output
+            })

.gitignore

@@ -4,6 +4,7 @@ queues.csv
 __pycache__/
 *.py[cod]
 
+.venv/
 venv/
 venv-freeze/
 
@@ -43,6 +44,7 @@ pip-delete-this-directory.txt
 htmlcov/
 .tox/
 .coverage
+.coverage_cache
 .coverage.*
 .cache
 .pytest_cache
@@ -56,6 +58,7 @@ test_results.xml
 
 # Django stuff:
 *.log
+/logs/*
 
 # Sphinx documentation
 docs/_build/
@@ -69,10 +72,23 @@ target/
 *.DS_Store
 environment.sh
 .envrc
+.env
+.env*
+varsfile
 
 celerybeat-schedule
+celerybeat-schedule.db
 
 # CloudFoundry
 .cf
+varsfile*
+.secret*
 
 /scripts/run_my_tests.sh
+
+# Terraform
+.terraform.lock.hcl
+**/.terraform/*
+secrets.auto.tfvars
+terraform.tfstate
+terraform.tfstate.backup

.profile

@@ -0,0 +1,7 @@
+##
+# Cloud Foundry app initialization script
+# https://docs.cloudfoundry.org/devguide/deploy-apps/deploy-app.html#profile
+##
+
+export https_proxy=$egress_proxy
+export NEW_RELIC_PROXY_HOST=$egress_proxy

.python-version

@@ -0,0 +1 @@
+3.9

.pyup.yml

@@ -1,8 +0,0 @@
-# see https://pyup.io/docs/configuration/ for all available options
-
-schedule: "every week on wednesday"
-
-search: False
-requirements:
-  - requirements.in
-  - requirements_for_test.txt

CONTRIBUTING.md

@@ -0,0 +1,19 @@
+# Welcome!
+
+We're so glad you're thinking about contributing to a [open source project of the U.S. government](https://code.gov/)! If you're unsure about anything, just ask -- or submit the issue or pull request anyway. The worst that can happen is you'll be politely asked to change something. We love all friendly contributions.
+
+We encourage you to read this project's CONTRIBUTING policy (you are here), its [LICENSE](LICENSE.md), and its [README](README.md).
+
+## Policies
+
+We want to ensure a welcoming environment for all of our projects. Our staff follow the [TTS Code of Conduct](https://18f.gsa.gov/code-of-conduct/) and all contributors should do the same.
+
+This project is adhering to the [18F Open Source Policy](https://github.com/18f/open-source-policy).
+
+As part of a U.S. government agency, the General Services Administration (GSA)’s Technology Transformation Services (TTS) takes seriously our responsibility to protect the public’s information, including financial and personal information, from unwarranted disclosure. For more information about security and vulnerability disclosure for our projects, please read our [18F Vulnerability Disclosure Policy](https://18f.gsa.gov/vulnerability-disclosure-policy/).
+
+## Public domain
+
+This project is in the public domain within the United States, and applicable copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+All contributions to this project will be released under the CC0 dedication. By submitting a pull request or issue, you are agreeing to comply with this waiver of copyright interest.

LICENSE

@@ -1,22 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2015 Government Digital Service
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-

LICENSE.md

@@ -0,0 +1,53 @@
+# License
+
+## A few parts of this project are not in the public domain
+
+### Copyright ownership of Works created by U.S. federal employees
+
+This repository contains both the original work for [GOV.UK Notify](https://github.com/alphagov/notifications-api) (the Software, as defined in the MIT License) and modifications made by the General Services Administration (GSA). This repository combines the Software and these GSA modifications, which is a “joint work” of the Software and modifications combined into a single work.
+
+As a work of the United States Government, the software modifications made by GSA are not subject to copyright within the United States. Additionally, GSA waives copyright and related rights in its software modifications worldwide through the [CC0 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+The Software remains subject to copyright under the MIT License. This consists of all work until [commit `e604385`](https://github.com/GSA/notifications-api/commit/e604385e0cf4c2ab8c6451b7120ceb196cce21b5) on June 1, 2022.
+
+#### Full license text for the MIT licensed files:
+
+```
+The MIT License (MIT)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+```
+
+## The rest of this project is in the worldwide public domain
+
+As a work of the United States government, this project is in the public domain within the United States.
+
+Additionally, we waive copyright and related rights in the work worldwide through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+
+### CC0 1.0 Universal Summary
+
+This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode).
+
+#### No copyright
+
+The person who associated a work with this deed has dedicated the work to the public domain by waiving all of his or her rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
+
+You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.
+
+#### Other information
+
+In no way are the patent or trademark rights of any person affected by CC0, nor are the rights that other persons may have in the work or in how the work is used, such as publicity or privacy rights.
+
+Unless expressly stated otherwise, the person who associated a work with this deed makes no warranties about the work, and disclaims liability for all uses of the work, to the fullest extent permitted by applicable law. When using or citing the work, you should not imply endorsement by the author or the affirmer.
+
+## Contributions to this project
+
+As stated in [CONTRIBUTING](CONTRIBUTING.md), all contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.
+
+## Trademark
+
+GSA owns, reserves, and retains all rights, title, and interest in all trademarks owned by GSA. The User does not acquire any right or interest of any kind in any GSA trademark because of their use of the Modified Version or any of the modifications made to the Work by GSA.

Makefile

@@ -7,55 +7,45 @@ APP_VERSION_FILE = app/version.py
 GIT_BRANCH ?= $(shell git symbolic-ref --short HEAD 2> /dev/null || echo "detached")
 GIT_COMMIT ?= $(shell git rev-parse HEAD)
 
-CF_API ?= api.cloud.service.gov.uk
-CF_ORG ?= govuk-notify
-CF_SPACE ?= ${DEPLOY_ENV}
-CF_HOME ?= ${HOME}
-$(eval export CF_HOME)
-
-CF_MANIFEST_PATH ?= /tmp/manifest.yml
-
-
-NOTIFY_CREDENTIALS ?= ~/.notify-credentials
-
-
 ## DEVELOPMENT
 
 .PHONY: bootstrap
-bootstrap: generate-version-file ## Set up everything to run the app
-	pip3 install -r requirements_for_test.txt
+bootstrap: ## Set up everything to run the app
+	make generate-version-file
+	pipenv install --dev
 	createdb notification_api || true
-	(. environment.sh && flask db upgrade) || true
+	(pipenv run flask db upgrade) || true
 
 .PHONY: bootstrap-with-docker
 bootstrap-with-docker: ## Build the image to run the app in Docker
 	docker build -f docker/Dockerfile -t notifications-api .
 
+.PHONY: run-procfile
+run-procfile:
+	pipenv run honcho start -f Procfile.dev
+
 .PHONY: run-flask
 run-flask: ## Run flask
-	. environment.sh && flask run -p 6011
+	pipenv run newrelic-admin run-program flask run -p 6011 --host=0.0.0.0
 
 .PHONY: run-celery
-run-celery: ## Run celery
-	. environment.sh && celery \
+run-celery: ## Run celery, TODO remove purge for staging/prod
+	pipenv run celery -A run_celery.notify_celery purge -f
+	pipenv run newrelic-admin run-program celery \
 		-A run_celery.notify_celery worker \
 		--pidfile="/tmp/celery.pid" \
 		--loglevel=INFO \
 		--concurrency=4
 
-.PHONY: run-celery-with-docker
-run-celery-with-docker: ## Run celery in Docker container (useful if you can't install pycurl locally)
-	./scripts/run_with_docker.sh make run-celery
-
 .PHONY: run-celery-beat
 run-celery-beat: ## Run celery beat
-	. environment.sh && celery \
-		-A run_celery.notify_celery beat \
-		--loglevel=INFO
+	pipenv run celery \
+	-A run_celery.notify_celery beat \
+	--loglevel=INFO
 
-.PHONY: run-celery-beat-with-docker
-run-celery-beat-with-docker: ## Run celery beat in Docker container (useful if you can't install pycurl locally)
-	./scripts/run_with_docker.sh make run-celery-beat
+.PHONY: cloudgov-user-report
+cloudgov-user-report:
+	@pipenv run python -m terraform.ops.cloudgov_user_report
 
 .PHONY: help
 help:
@@ -66,15 +56,29 @@ generate-version-file: ## Generates the app version file
 	@echo -e "__git_commit__ = \"${GIT_COMMIT}\"\n__time__ = \"${DATE}\"" > ${APP_VERSION_FILE}
 
 .PHONY: test
-test: ## Run tests
-	flake8 .
-	isort --check-only ./app ./tests
-	pytest -n4 --maxfail=10
+test: export NEW_RELIC_ENVIRONMENT=test
+test: ## Run tests and create coverage report
+	pipenv run flake8 .
+	pipenv run isort --check-only ./app ./tests
+	pipenv run coverage run --omit=*/notifications_utils/* -m pytest --maxfail=10
+	pipenv run coverage report --fail-under=50
+	pipenv run coverage html -d .coverage_cache
 
 .PHONY: freeze-requirements
 freeze-requirements: ## Pin all requirements including sub dependencies into requirements.txt
-	pip install --upgrade pip-tools
-	pip-compile requirements.in
+	pipenv lock
+	pipenv requirements
+
+.PHONY: audit
+audit:
+	pipenv requirements > requirements.txt
+	pipenv requirements --dev > requirements_for_test.txt
+	pipenv run pip-audit -r requirements.txt
+	-pipenv run pip-audit -r requirements_for_test.txt
+
+.PHONY: static-scan
+static-scan:
+	pipenv run bandit -r app/
 
 .PHONY: clean
 clean:
@@ -83,109 +87,26 @@ clean:
 
 ## DEPLOYMENT
 
-.PHONY: preview
-preview: ## Set environment to preview
-	$(eval export DEPLOY_ENV=preview)
-	$(eval export DNS_NAME="notify.works")
-	@true
+# .PHONY: cf-deploy-failwhale
+# cf-deploy-failwhale:
+# 	$(if ${CF_SPACE},,$(error Must target space, eg `make preview cf-deploy-failwhale`))
+# 	cd ./paas-failwhale; cf push notify-api-failwhale -f manifest.yml
 
-.PHONY: staging
-staging: ## Set environment to staging
-	$(eval export DEPLOY_ENV=staging)
-	$(eval export DNS_NAME="staging-notify.works")
-	@true
+# .PHONY: enable-failwhale
+# enable-failwhale: ## Enable the failwhale app and disable api
+# 	$(if ${DNS_NAME},,$(error Must target space, eg `make preview enable-failwhale`))
+# 	# make sure failwhale is running first
+# 	cf start notify-api-failwhale
 
-.PHONY: production
-production: ## Set environment to production
-	$(eval export DEPLOY_ENV=production)
-	$(eval export DNS_NAME="notifications.service.gov.uk")
-	@true
+# 	cf map-route notify-api-failwhale ${DNS_NAME} --hostname api
+# 	cf unmap-route notify-api ${DNS_NAME} --hostname api
+# 	@echo "Failwhale is enabled"
 
-.PHONY: cf-login
-cf-login: ## Log in to Cloud Foundry
-	$(if ${CF_USERNAME},,$(error Must specify CF_USERNAME))
-	$(if ${CF_PASSWORD},,$(error Must specify CF_PASSWORD))
-	$(if ${CF_SPACE},,$(error Must specify CF_SPACE))
-	@echo "Logging in to Cloud Foundry on ${CF_API}"
-	@cf login -a "${CF_API}" -u ${CF_USERNAME} -p "${CF_PASSWORD}" -o "${CF_ORG}" -s "${CF_SPACE}"
+# .PHONY: disable-failwhale
+# disable-failwhale: ## Disable the failwhale app and enable api
+# 	$(if ${DNS_NAME},,$(error Must target space, eg `make preview disable-failwhale`))
 
-.PHONY: generate-manifest
-generate-manifest:
-	$(if ${CF_APP},,$(error Must specify CF_APP))
-	$(if ${CF_SPACE},,$(error Must specify CF_SPACE))
-	$(if $(shell which gpg2), $(eval export GPG=gpg2), $(eval export GPG=gpg))
-	$(if ${GPG_PASSPHRASE_TXT}, $(eval export DECRYPT_CMD=echo -n $$$${GPG_PASSPHRASE_TXT} | ${GPG} --quiet --batch --passphrase-fd 0 --pinentry-mode loopback -d), $(eval export DECRYPT_CMD=${GPG} --quiet --batch -d))
-
-	@jinja2 --strict manifest.yml.j2 \
-	    -D environment=${CF_SPACE} \
-	    -D CF_APP=${CF_APP} \
-	    --format=yaml \
-	    <(${DECRYPT_CMD} ${NOTIFY_CREDENTIALS}/credentials/${CF_SPACE}/paas/environment-variables.gpg) 2>&1
-
-.PHONY: cf-deploy
-cf-deploy: ## Deploys the app to Cloud Foundry
-	$(if ${CF_SPACE},,$(error Must specify CF_SPACE))
-	$(if ${CF_APP},,$(error Must specify CF_APP))
-	cf target -o ${CF_ORG} -s ${CF_SPACE}
-	@cf app --guid ${CF_APP} || exit 1
-
-	# cancel any existing deploys to ensure we can apply manifest (if a deploy is in progress you'll see ScaleDisabledDuringDeployment)
-	cf cancel-deployment ${CF_APP} || true
-
-	# generate manifest (including secrets) and write it to CF_MANIFEST_PATH (in /tmp/)
-	make -s CF_APP=${CF_APP} generate-manifest > ${CF_MANIFEST_PATH}
-
-	# fails after 15 mins if deploy doesn't work
-	# reads manifest from CF_MANIFEST_PATH
-	CF_STARTUP_TIMEOUT=15 cf push ${CF_APP} --strategy=rolling -f ${CF_MANIFEST_PATH}
-	# delete old manifest file
-	rm ${CF_MANIFEST_PATH}
-
-.PHONY: cf-deploy-api-db-migration
-cf-deploy-api-db-migration:
-	$(if ${CF_SPACE},,$(error Must specify CF_SPACE))
-	cf target -o ${CF_ORG} -s ${CF_SPACE}
-	make -s CF_APP=notify-api-db-migration generate-manifest > ${CF_MANIFEST_PATH}
-
-	cf push notify-api-db-migration --no-route -f ${CF_MANIFEST_PATH}
-	rm ${CF_MANIFEST_PATH}
-
-	cf run-task notify-api-db-migration --command="flask db upgrade" --name api_db_migration
-
-.PHONY: cf-check-api-db-migration-task
-cf-check-api-db-migration-task: ## Get the status for the last notify-api-db-migration task
-	@cf curl /v3/apps/`cf app --guid notify-api-db-migration`/tasks?order_by=-created_at | jq -r ".resources[0].state"
-
-.PHONY: cf-rollback
-cf-rollback: ## Rollbacks the app to the previous release
-	$(if ${CF_APP},,$(error Must specify CF_APP))
-	rm ${CF_MANIFEST_PATH}
-	cf cancel-deployment ${CF_APP}
-
-.PHONY: check-if-migrations-to-run
-check-if-migrations-to-run:
-	@echo $(shell python3 scripts/check_if_new_migration.py)
-
-.PHONY: cf-deploy-failwhale
-cf-deploy-failwhale:
-	$(if ${CF_SPACE},,$(error Must target space, eg `make preview cf-deploy-failwhale`))
-	cd ./paas-failwhale; cf push notify-api-failwhale -f manifest.yml
-
-.PHONY: enable-failwhale
-enable-failwhale: ## Enable the failwhale app and disable api
-	$(if ${DNS_NAME},,$(error Must target space, eg `make preview enable-failwhale`))
-	# make sure failwhale is running first
-	cf start notify-api-failwhale
-
-	cf map-route notify-api-failwhale ${DNS_NAME} --hostname api
-	cf unmap-route notify-api ${DNS_NAME} --hostname api
-	@echo "Failwhale is enabled"
-
-.PHONY: disable-failwhale
-disable-failwhale: ## Disable the failwhale app and enable api
-	$(if ${DNS_NAME},,$(error Must target space, eg `make preview disable-failwhale`))
-
-	cf map-route notify-api ${DNS_NAME} --hostname api
-	cf unmap-route notify-api-failwhale ${DNS_NAME} --hostname api
-	cf stop notify-api-failwhale
-	@echo "Failwhale is disabled"
+# 	cf map-route notify-api ${DNS_NAME} --hostname api
+# 	cf unmap-route notify-api-failwhale ${DNS_NAME} --hostname api
+# 	cf stop notify-api-failwhale
+# 	@echo "Failwhale is disabled"

Pipfile

@@ -0,0 +1,84 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+alembic = "==1.7.7"
+amqp = "==5.1.1"
+arrow = "==1.2.2"
+asn1crypto = "==1.5.1"
+async-timeout = "==4.0.2"
+attrs = "==21.4.0"
+awscli = "==1.24.8"
+bcrypt = "==3.2.2"
+beautifulsoup4 = "==4.12.2"
+billiard = "==3.6.4.0"
+bleach = "==4.1.0"
+blinker = "~=1.4"
+boto3 = "==1.23.8"
+botocore = "==1.26.8"
+cachetools = "==5.1.0"
+celery = {version = "==5.2.7", extras = ["redis"]}
+certifi = ">=2022.12.7"
+cffi = "==1.15.0"
+charset-normalizer = "==2.0.12"
+click = "==8.1.3"
+click-datetime = "==0.2"
+click-didyoumean = "==0.3.0"
+click-plugins = "==1.1.1"
+click-repl = "==0.2.0"
+colorama = "==0.4.4"
+defusedxml = "==0.7.1"
+deprecated = "==1.2.13"
+dnspython = "==2.2.1"
+docopt = "==0.6.2"
+docutils = "==0.16"
+eventlet = "==0.33.1"
+flask = "~=2.3"
+flask-bcrypt = "==1.0.1"
+flask-marshmallow = "==0.14.0"
+flask-migrate = "==3.1.0"
+flask-redis = "==0.4.0"
+flask-sqlalchemy = {version = "==2.5.1", ref = "aa7a61a5357cf6f5dcc135d98c781192457aa6fa", git = "https://github.com/pallets-eco/flask-sqlalchemy.git"}
+gunicorn = {version = "==20.1.0", extras = ["eventlet"], ref = "1299ea9e967a61ae2edebe191082fd169b864c64", git = "https://github.com/benoitc/gunicorn.git"}
+iso8601 = "==1.0.2"
+itsdangerous = "==2.1.2"
+jsonschema = {version = "==4.17.3", extras = ["format"]}
+lxml = "==4.9.1"
+marshmallow = "==3.19.0"
+marshmallow-sqlalchemy = "==0.29.0"
+notifications-python-client = "==6.3.0"
+oscrypto = "==1.3.0"
+psycopg2-binary = "==2.9.3"
+pyjwt = "==2.7.0"
+python-dotenv = "==1.0.0"
+sqlalchemy = "==1.4.40"
+werkzeug = "~=2.3"
+# gds metrics packages
+prometheus-client = "==0.14.1"
+gds-metrics = {version = "==0.2.4", ref = "6f1840a57b6fb1ee40b7e84f2f18ec229de8aa72", git = "https://github.com/alphagov/gds_metrics_python.git"}
+packaging = "==23.1"
+notifications-utils = {editable = true, ref = "main", git = "https://github.com/GSA/notifications-utils.git"}
+newrelic = "*"
+
+[dev-packages]
+flake8 = "==4.0.1"
+flake8-bugbear = "==22.4.25"
+isort = "==5.10.1"
+moto = "==3.1.9"
+pytest = "==7.1.2"
+pytest-env = "==0.6.2"
+pytest-mock = "==3.11.1"
+pytest-cov = "==3.0.0"
+pytest-xdist = "==2.5.0"
+freezegun = "==1.2.1"
+requests-mock = "==1.9.3"
+jinja2-cli = {version = "==0.8.2", extras = ["yaml"]}
+pip-audit = "*"
+bandit = "*"
+honcho = "*"
+cloudfoundry-client = "*"
+
+[requires]
+python_version = "3.9"

Procfile

@@ -1 +0,0 @@
-web: ./scripts/paas_app_wrapper.sh

Procfile.dev

@@ -0,0 +1,3 @@
+web: make run-flask
+worker: make run-celery
+scheduler: make run-celery-beat

README.md

@@ -1,139 +1,132 @@
-# GOV.UK Notify API
+# US Notify API
 
-Contains:
-- the public-facing REST API for GOV.UK Notify, which teams can integrate with using [our clients](https://www.notifications.service.gov.uk/documentation)
-- an internal-only REST API built using Flask to manage services, users, templates, etc (this is what the [admin app](http://github.com/alphagov/notifications-admin) talks to)
-- asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc
+This project is the core of [Notify](https://notifications-admin.app.cloud.gov/). It's cloned from the brilliant work of the team at [GOV.UK Notify](https://github.com/alphagov/notifications-api), cheers!
 
-## Setting Up
+This repo contains:
 
-### Python version
+- A public-facing REST API for Notify, which teams can integrate with using [API clients built by UK](https://www.notifications.service.gov.uk/documentation)
+- An internal-only REST API built using Flask to manage services, users, templates, etc., which the [admin UI](http://github.com/18F/notifications-admin) talks to)
+- Asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc.
 
-We run python 3.9 both locally and in production.
+Our other repositories are:
 
-### psycopg2
+- [notifications-admin](https://github.com/GSA/notifications-admin)
+- [notifications-utils](https://github.com/GSA/notifications-utils)
+- [us-notify-compliance](https://github.com/GSA/us-notify-compliance/)
+- [notify-python-demo](https://github.com/GSA/notify-python-demo)
 
-[Follow these instructions on Mac M1 machines](https://github.com/psycopg/psycopg2/issues/1216#issuecomment-1068150544).
+## Documentation, here and elsewhere
 
-### AWS credentials
+### About Notify
 
-To run the API you will need appropriate AWS credentials. See the [Wiki](https://github.com/alphagov/notifications-manuals/wiki/aws-accounts#how-to-set-up-local-development) for more details.
+- [Roadmap](https://notifications-admin.app.cloud.gov/features/roadmap)
+- [Using the API](./docs/api-usage.md)
+- [Architectural Decision Records](./docs/adrs/)
 
-### `environment.sh`
+### Infrastructure
 
-Creating and edit an environment.sh file.
+- [Overview, setup, and onboarding](./docs/infra-overview.md)
+- [Database management](./docs/database-management.md)
+- [Celery queues and tasks](./docs/queues-and-tasks.md)
 
-```
-echo "
-export NOTIFY_ENVIRONMENT='development'
+### Common dev work
 
-export MMG_API_KEY='MMG_API_KEY'
-export FIRETEXT_API_KEY='FIRETEXT_ACTUAL_KEY'
-export REACH_API_KEY='REACH_API_KEY'
-export NOTIFICATION_QUEUE_PREFIX='YOUR_OWN_PREFIX'
+- [Local setup](#local-setup)
+- [Testing](./docs/testing.md), both automated and manual
+- [Deploying](./docs/deploying.md)
+- [Running one-off tasks](./docs/one-off-tasks.md)
 
-export FLASK_APP=application.py
-export FLASK_ENV=development
-export WERKZEUG_DEBUG_PIN=off
-"> environment.sh
-```
-
-Things to change:
-
-* Replace `YOUR_OWN_PREFIX` with `local_dev_<first name>`.
-* Run the following in the credentials repo to get the API keys.
-
-```
-notify-pass credentials/firetext
-notify-pass credentials/mmg
-notify-pass credentials/reach
-```
-
-### Postgres
-
-Install [Postgres.app](http://postgresapp.com/).
-
-Currently the API works with PostgreSQL 11. After installation, open the Postgres app, open the sidebar, and update or replace the default server with a compatible version.
-
-**Note:** you may need to add the following directory to your PATH in order to bootstrap the app.
-
-```
-export PATH=${PATH}:/Applications/Postgres.app/Contents/Versions/11/bin/
-```
-
-### Redis
-
-To switch redis on you'll need to install it locally. On a Mac you can do:
-
-```
-# assuming you use Homebrew
-brew install redis
-brew services start redis
-```
-
-To use redis caching you need to switch it on with an environment variable:
-
-```
-export REDIS_ENABLED=1
-```
-
-##  To run the application
-
-```
-# install dependencies, etc.
-make bootstrap
-
-# run the web app
-make run-flask
-
-# run the background tasks
-make run-celery
-
-# run scheduled tasks (optional)
-make run-celery-beat
-```
-
-We've had problems running Celery locally due to one of its dependencies: pycurl. Due to the complexity of the issue, we also support running Celery via Docker:
-
-```
-# install dependencies, etc.
-make bootstrap-with-docker
-
-# run the background tasks
-make run-celery-with-docker
-
-# run scheduled tasks
-make run-celery-beat-with-docker
-```
-
-##  To test the application
-
-```
-# install dependencies, etc.
-make bootstrap
-
-make test
-```
-
-## To run one off tasks
-
-Tasks are run through the `flask` command - run `flask --help` for more information. There are two sections we need to
-care about: `flask db` contains alembic migration commands, and `flask command` contains all of our custom commands. For
-example, to purge all dynamically generated functional test data, do the following:
-
-Locally
-```
-flask command purge_functional_test_data -u <functional tests user name prefix>
-```
-
-On the server
-```
-cf run-task notify-api "flask command purge_functional_test_data -u <functional tests user name prefix>"
-```
-
-All commands and command options have a --help command if you need more information.
-
-## Further documentation
+## UK docs that may still be helpful
 
 - [Writing public APIs](docs/writing-public-apis.md)
 - [Updating dependencies](https://github.com/alphagov/notifications-manuals/wiki/Dependencies)
+
+## Local setup
+
+### Common steps
+
+On MacOS, using [Homebrew](https://brew.sh/) for package management is highly recommended. This helps avoid some known installation issues.
+
+1. Install pre-requisites for setup:
+    * [jq](https://stedolan.github.io/jq/): `brew install jq`
+    * [terraform](https://www.terraform.io/): `brew install terraform` or `brew install tfenv` and use `tfenv` to install `terraform ~> 1.4.0`
+    * [cf-cli@8](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html): `brew install cloudfoundry/tap/cf-cli@8`
+    * [postgresql](https://www.postgresql.org/): `brew install postgresql@15` (Homebrew requires a version pin, but any recent version will work)
+    * [redis](https://redis.io/): `brew install redis`
+    * [pyenv](https://github.com/pyenv/pyenv): `brew install pyenv`
+1. [Log into cloud.gov](https://cloud.gov/docs/getting-started/setup/#set-up-the-command-line): `cf login -a api.fr.cloud.gov --sso`
+1. Ensure you have access to the `notify-local-dev` and `notify-staging` spaces in cloud.gov
+1. Run the development terraform with:
+
+        ```
+        $ cd terraform/development
+        $ ./run.sh
+        ```
+
+1. If you want to send data to New Relic from your local develpment environment, set `NEW_RELIC_LICENSE_KEY` within `.env`
+1. Follow the instructions for either `Direct installation` or `Docker installation` below
+
+### Direct installation
+
+1. Set up Postgres && Redis on your machine
+
+1. Install [pipenv](https://pipenv.pypa.io/en/latest/)
+
+1. Run the project setup
+
+    `make bootstrap`
+
+1. Run the web server and background worker
+
+    `make run-procfile`
+
+1. Or run them individually:
+
+    * Run Flask (web server)
+
+        `make run-flask`
+
+    * Run Celery (background worker)
+
+        `make run-celery`
+
+
+### VS Code && Docker installation
+
+If you're working in VS Code, you can also leverage Docker for a containerized dev environment
+
+1. Uncomment the `Local Docker setup` lines in `.env` and comment out the `Local direct setup` lines.
+
+1. Install the Remote-Containers plug-in in VS Code
+
+1. With Docker running, create the network:
+
+    `docker network create notify-network`
+
+1. Using the command palette (shift+cmd+p) or green button thingy in the bottom left, search and select “Remote Containers: Open Folder in Container...” When prompted, choose **devcontainer-api** folder (note: this is a *subfolder* of notifications-api). This will start the container in a new window, replacing the current one.
+
+1. Wait a few minutes while things happen 🍵
+
+1. Open a VS Code terminal and run the Flask application:
+
+    `make run-flask`
+
+1. Open another VS Code terminal and run Celery:
+
+    `make run-celery`
+
+NOTE: when you change .env in the future, you'll need to rebuild the devcontainer for the change to take effect. VS Code _should_ detect the change and prompt you with a toast notification during a cached build. If not, you can find a manual rebuild in command pallette or just `docker rm` the notifications-api container.
+
+### Known installation issues
+
+On M1 Macs, if you get a `fatal error: 'Python.h' file not found` message, try a different method of installing Python. Installation via `pyenv` is known to work.
+
+A direct installation of PostgreSQL will not put the `createdb` command on your `$PATH`. It can be added there in your shell startup script, or a Homebrew-managed installation of PostgreSQL will take care of it.
+
+## License && public domain
+
+Work through [commit `e604385`](https://github.com/GSA/notifications-api/commit/e604385e0cf4c2ab8c6451b7120ceb196cce21b5) is licensed by the UK government under the MIT license. Work after that commit is in the worldwide public domain. See [LICENSE.md](./LICENSE.md) for more information.
+
+## Contributing
+
+As stated in [CONTRIBUTING.md](CONTRIBUTING.md), all contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

app/__init__.py

@@ -1,5 +1,5 @@
 import os
-import random
+import secrets
 import string
 import time
 import uuid
@@ -23,48 +23,43 @@ from notifications_utils import logging, request_helper
 from notifications_utils.celery import NotifyCelery
 from notifications_utils.clients.encryption.encryption_client import Encryption
 from notifications_utils.clients.redis.redis_client import RedisClient
-from notifications_utils.clients.statsd.statsd_client import StatsdClient
 from notifications_utils.clients.zendesk.zendesk_client import ZendeskClient
 from sqlalchemy import event
 from werkzeug.exceptions import HTTPException as WerkzeugHTTPException
 from werkzeug.local import LocalProxy
 
 from app.clients import NotificationProviderClients
-from app.clients.cbc_proxy import CBCProxyClient
+from app.clients.cloudwatch.aws_cloudwatch import AwsCloudwatchClient
 from app.clients.document_download import DocumentDownloadClient
 from app.clients.email.aws_ses import AwsSesClient
 from app.clients.email.aws_ses_stub import AwsSesStubClient
-from app.clients.sms.firetext import FiretextClient
-from app.clients.sms.mmg import MMGClient
-from app.clients.sms.reach import ReachClient
+from app.clients.sms.aws_sns import AwsSnsClient
 
 
 class SQLAlchemy(_SQLAlchemy):
     """We need to subclass SQLAlchemy in order to override create_engine options"""
 
     def apply_driver_hacks(self, app, info, options):
-        super().apply_driver_hacks(app, info, options)
+        sa_url, options = super().apply_driver_hacks(app, info, options)
         if 'connect_args' not in options:
             options['connect_args'] = {}
         options['connect_args']["options"] = "-c statement_timeout={}".format(
             int(app.config['SQLALCHEMY_STATEMENT_TIMEOUT']) * 1000
         )
+        return (sa_url, options)
 
 
 db = SQLAlchemy()
 migrate = Migrate()
 ma = Marshmallow()
 notify_celery = NotifyCelery()
-firetext_client = FiretextClient()
-mmg_client = MMGClient()
-reach_client = ReachClient()
 aws_ses_client = AwsSesClient()
 aws_ses_stub_client = AwsSesStubClient()
+aws_sns_client = AwsSnsClient()
+aws_cloudwatch_client = AwsCloudwatchClient()
 encryption = Encryption()
 zendesk_client = ZendeskClient()
-statsd_client = StatsdClient()
 redis_store = RedisClient()
-cbc_proxy_client = CBCProxyClient()
 document_download_client = DocumentDownloadClient()
 metrics = GDSMetrics()
 
@@ -96,22 +91,18 @@ def create_app(application):
     migrate.init_app(application, db=db)
     ma.init_app(application)
     zendesk_client.init_app(application)
-    statsd_client.init_app(application)
-    logging.init_app(application, statsd_client)
-    firetext_client.init_app(application, statsd_client=statsd_client)
-    mmg_client.init_app(application, statsd_client=statsd_client)
-    reach_client.init_app(application, statsd_client=statsd_client)
+    logging.init_app(application)
+    aws_sns_client.init_app(application)
 
-    aws_ses_client.init_app(application.config['AWS_REGION'], statsd_client=statsd_client)
+    aws_ses_client.init_app()
     aws_ses_stub_client.init_app(
-        application.config['AWS_REGION'],
-        statsd_client=statsd_client,
         stub_url=application.config['SES_STUB_URL']
     )
+    aws_cloudwatch_client.init_app(application)
     # If a stub url is provided for SES, then use the stub client rather than the real SES boto client
     email_clients = [aws_ses_stub_client] if application.config['SES_STUB_URL'] else [aws_ses_client]
     notification_provider_clients.init_app(
-        sms_clients=[firetext_client, mmg_client, reach_client],
+        sms_clients=[aws_sns_client],
         email_clients=email_clients
     )
 
@@ -120,8 +111,6 @@ def create_app(application):
     redis_store.init_app(application)
     document_download_client.init_app(application)
 
-    cbc_proxy_client.init_app(application)
-
     register_blueprint(application)
     register_v2_blueprints(application)
 
@@ -139,34 +128,25 @@ def register_blueprint(application):
     from app.authentication.auth import (
         requires_admin_auth,
         requires_auth,
-        requires_govuk_alerts_auth,
         requires_no_auth,
     )
     from app.billing.rest import billing_blueprint
-    from app.broadcast_message.rest import broadcast_message_blueprint
     from app.complaint.complaint_rest import complaint_blueprint
+    from app.docs import docs as docs_blueprint
     from app.email_branding.rest import email_branding_blueprint
     from app.events.rest import events as events_blueprint
-    from app.govuk_alerts.rest import govuk_alerts_blueprint
     from app.inbound_number.rest import inbound_number_blueprint
     from app.inbound_sms.rest import inbound_sms as inbound_sms_blueprint
     from app.job.rest import job_blueprint
-    from app.letter_branding.letter_branding_rest import (
-        letter_branding_blueprint,
-    )
-    from app.letters.rest import letter_job
-    from app.notifications.notifications_letter_callback import (
-        letter_callback_blueprint,
-    )
-    from app.notifications.notifications_sms_callback import (
-        sms_callback_blueprint,
+    from app.notifications.notifications_ses_callback import (
+        ses_callback_blueprint,
     )
     from app.notifications.receive_notifications import (
         receive_notifications_blueprint,
     )
     from app.notifications.rest import notifications as notifications_blueprint
-    from app.organisation.invite_rest import organisation_invite_blueprint
-    from app.organisation.rest import organisation_blueprint
+    from app.organization.invite_rest import organization_invite_blueprint
+    from app.organization.rest import organization_blueprint
     from app.performance_dashboard.rest import performance_dashboard_blueprint
     from app.platform_stats.rest import platform_stats_blueprint
     from app.provider_details.rest import (
@@ -202,10 +182,12 @@ def register_blueprint(application):
     status_blueprint.before_request(requires_no_auth)
     application.register_blueprint(status_blueprint)
 
+    docs_blueprint.before_request(requires_no_auth)
+    application.register_blueprint(docs_blueprint)
+
     # delivery receipts
-    # TODO: make sure research mode can still trigger sms callbacks, then re-enable this
-    sms_callback_blueprint.before_request(requires_no_auth)
-    application.register_blueprint(sms_callback_blueprint)
+    ses_callback_blueprint.before_request(requires_no_auth)
+    application.register_blueprint(ses_callback_blueprint)
 
     # inbound sms
     receive_notifications_blueprint.before_request(requires_no_auth)
@@ -220,8 +202,8 @@ def register_blueprint(application):
     service_invite_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(service_invite_blueprint)
 
-    organisation_invite_blueprint.before_request(requires_admin_auth)
-    application.register_blueprint(organisation_invite_blueprint)
+    organization_invite_blueprint.before_request(requires_admin_auth)
+    application.register_blueprint(organization_invite_blueprint)
 
     inbound_number_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(inbound_number_blueprint)
@@ -241,20 +223,14 @@ def register_blueprint(application):
     email_branding_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(email_branding_blueprint, url_prefix='/email-branding')
 
-    letter_job.before_request(requires_admin_auth)
-    application.register_blueprint(letter_job)
-
-    letter_callback_blueprint.before_request(requires_no_auth)
-    application.register_blueprint(letter_callback_blueprint)
-
     billing_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(billing_blueprint)
 
     service_callback_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(service_callback_blueprint)
 
-    organisation_blueprint.before_request(requires_admin_auth)
-    application.register_blueprint(organisation_blueprint, url_prefix='/organisations')
+    organization_blueprint.before_request(requires_admin_auth)
+    application.register_blueprint(organization_blueprint, url_prefix='/organizations')
 
     complaint_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(complaint_blueprint)
@@ -268,22 +244,12 @@ def register_blueprint(application):
     template_folder_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(template_folder_blueprint)
 
-    letter_branding_blueprint.before_request(requires_admin_auth)
-    application.register_blueprint(letter_branding_blueprint)
-
     upload_blueprint.before_request(requires_admin_auth)
     application.register_blueprint(upload_blueprint)
 
-    broadcast_message_blueprint.before_request(requires_admin_auth)
-    application.register_blueprint(broadcast_message_blueprint)
-
-    govuk_alerts_blueprint.before_request(requires_govuk_alerts_auth)
-    application.register_blueprint(govuk_alerts_blueprint)
-
 
 def register_v2_blueprints(application):
     from app.authentication.auth import requires_auth
-    from app.v2.broadcast.post_broadcast import v2_broadcast_blueprint
     from app.v2.inbound_sms.get_inbound_sms import v2_inbound_sms_blueprint
     from app.v2.notifications import (  # noqa
         get_notifications,
@@ -309,9 +275,6 @@ def register_v2_blueprints(application):
     v2_inbound_sms_blueprint.before_request(requires_auth)
     application.register_blueprint(v2_inbound_sms_blueprint)
 
-    v2_broadcast_blueprint.before_request(requires_auth)
-    application.register_blueprint(v2_broadcast_blueprint)
-
 
 def init_app(app):
 
@@ -326,9 +289,7 @@ def init_app(app):
     def after_request(response):
         CONCURRENT_REQUESTS.dec()
 
-        response.headers.add('Access-Control-Allow-Origin', '*')
-        response.headers.add('Access-Control-Allow-Headers', 'Content-Type,Authorization')
-        response.headers.add('Access-Control-Allow-Methods', 'GET,PUT,POST,DELETE')
+        response.headers.add('X-Content-Type-Options', 'nosniff')
         return response
 
     @app.errorhandler(Exception)
@@ -337,20 +298,34 @@ def init_app(app):
         # error.code is set for our exception types.
         msg = getattr(error, 'message', str(error))
         code = getattr(error, 'code', 500)
-        return jsonify(result='error', message=msg), code
+        response = make_response(
+            jsonify(result='error', message=msg),
+            code,
+            error.get_headers()
+        )
+        response.content_type = "application/json"
+        return response
 
     @app.errorhandler(WerkzeugHTTPException)
     def werkzeug_exception(e):
-        return make_response(
+        response = make_response(
             jsonify(result='error', message=e.description),
             e.code,
             e.get_headers()
         )
+        response.content_type = 'application/json'
+        return response
 
     @app.errorhandler(404)
     def page_not_found(e):
         msg = e.description or "Not found"
-        return jsonify(result='error', message=msg), 404
+        response = make_response(
+            jsonify(result='error', message=msg),
+            404,
+            e.get_headers()
+        )
+        response.content_type = 'application/json'
+        return response
 
 
 def create_uuid():
@@ -358,7 +333,7 @@ def create_uuid():
 
 
 def create_random_identifier():
-    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16))
+    return ''.join(secrets.choice(string.ascii_uppercase + string.digits) for _ in range(16))
 
 
 def setup_sqlalchemy_events(app):

app/authentication/auth.py

@@ -18,7 +18,11 @@ from sqlalchemy.orm.exc import NoResultFound
 
 from app.serialised_models import SerialisedService
 
-GENERAL_TOKEN_ERROR_MESSAGE = 'Invalid token: make sure your API token matches the example at https://docs.notifications.service.gov.uk/rest-api.html#authorisation-header'  # noqa
+# stvnrlly - this is silly, but bandit has a multiline string bug (https://github.com/PyCQA/bandit/issues/658)
+# and flake8 wants a multiline quote here. TODO: check on bug status and restore sanity once possible
+TOKEN_MESSAGE_ONE = "Invalid token: make sure your API token matches the example "  # nosec B105
+TOKEN_MESSAGE_TWO = "at https://docs.notifications.service.gov.uk/rest-api.html#authorisation-header"  # nosec B105
+GENERAL_TOKEN_ERROR_MESSAGE = TOKEN_MESSAGE_ONE + TOKEN_MESSAGE_TWO
 
 AUTH_DB_CONNECTION_DURATION_SECONDS = Histogram(
     'auth_db_connection_duration_seconds',
@@ -60,10 +64,6 @@ def requires_no_auth():
     pass
 
 
-def requires_govuk_alerts_auth():
-    requires_internal_auth(current_app.config.get('GOVUK_ALERTS_CLIENT_ID'))
-
-
 def requires_admin_auth():
     requires_internal_auth(current_app.config.get('ADMIN_CLIENT_ID'))
 
@@ -77,6 +77,8 @@ def requires_internal_auth(expected_client_id):
     client_id = _get_token_issuer(auth_token)
 
     if client_id != expected_client_id:
+        current_app.logger.info('client_id: %s', client_id)
+        current_app.logger.info('expected_client_id: %s', expected_client_id)
         raise AuthError("Unauthorized: not allowed to perform this action", 401)
 
     api_keys = [
@@ -130,8 +132,9 @@ def _decode_jwt_token(auth_token, api_keys, service_id=None):
         try:
             decode_jwt_token(auth_token, api_key.secret)
         except TokenExpiredError:
-            err_msg = "Error: Your system clock must be accurate to within 30 seconds"
-            raise AuthError(err_msg, 403, service_id=service_id, api_key_id=api_key.id)
+            if not current_app.config.get('ALLOW_EXPIRED_API_TOKEN', False):
+                err_msg = "Error: Your system clock must be accurate to within 30 seconds"
+                raise AuthError(err_msg, 403, service_id=service_id, api_key_id=api_key.id)
         except TokenAlgorithmError:
             err_msg = "Invalid token: algorithm used is not HS256"
             raise AuthError(err_msg, 403, service_id=service_id, api_key_id=api_key.id)

app/aws/s3.py

@@ -1,30 +1,31 @@
 import botocore
-from boto3 import client, resource
+from boto3 import Session
 from flask import current_app
 
 FILE_LOCATION_STRUCTURE = 'service-{}-notify/{}.csv'
 
 
-def get_s3_file(bucket_name, file_location):
-    s3_file = get_s3_object(bucket_name, file_location)
+def get_s3_file(
+    bucket_name, file_location, access_key, secret_key, region
+):
+    s3_file = get_s3_object(bucket_name, file_location, access_key, secret_key, region)
     return s3_file.get()['Body'].read().decode('utf-8')
 
 
-def get_s3_object(bucket_name, file_location):
-    s3 = resource('s3')
+def get_s3_object(
+    bucket_name, file_location, access_key, secret_key, region
+):
+    session = Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=region)
+    s3 = session.resource('s3')
     return s3.Object(bucket_name, file_location)
 
 
-def head_s3_object(bucket_name, file_location):
-    # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object
-    boto_client = client('s3', current_app.config['AWS_REGION'])
-    return boto_client.head_object(Bucket=bucket_name, Key=file_location)
-
-
-def file_exists(bucket_name, file_location):
+def file_exists(
+    bucket_name, file_location, access_key, secret_key, region
+):
     try:
         # try and access metadata of object
-        get_s3_object(bucket_name, file_location).metadata
+        get_s3_object(bucket_name, file_location, access_key, secret_key, region).metadata
         return True
     except botocore.exceptions.ClientError as e:
         if e.response['ResponseMetadata']['HTTPStatusCode'] == 404:
@@ -34,15 +35,11 @@ def file_exists(bucket_name, file_location):
 
 def get_job_location(service_id, job_id):
     return (
-        current_app.config['CSV_UPLOAD_BUCKET_NAME'],
+        current_app.config['CSV_UPLOAD_BUCKET']['bucket'],
         FILE_LOCATION_STRUCTURE.format(service_id, job_id),
-    )
-
-
-def get_contact_list_location(service_id, contact_list_id):
-    return (
-        current_app.config['CONTACT_LIST_BUCKET_NAME'],
-        FILE_LOCATION_STRUCTURE.format(service_id, contact_list_id),
+        current_app.config['CSV_UPLOAD_BUCKET']['access_key_id'],
+        current_app.config['CSV_UPLOAD_BUCKET']['secret_access_key'],
+        current_app.config['CSV_UPLOAD_BUCKET']['region'],
     )
 
 
@@ -65,27 +62,17 @@ def remove_job_from_s3(service_id, job_id):
     return remove_s3_object(*get_job_location(service_id, job_id))
 
 
-def remove_contact_list_from_s3(service_id, contact_list_id):
-    return remove_s3_object(*get_contact_list_location(service_id, contact_list_id))
-
-
-def remove_s3_object(bucket_name, object_key):
-    obj = get_s3_object(bucket_name, object_key)
+def remove_s3_object(bucket_name, object_key, access_key, secret_key, region):
+    obj = get_s3_object(bucket_name, object_key, access_key, secret_key, region)
     return obj.delete()
 
 
-def get_list_of_files_by_suffix(bucket_name, subfolder='', suffix='', last_modified=None):
-    s3_client = client('s3', current_app.config['AWS_REGION'])
-    paginator = s3_client.get_paginator('list_objects_v2')
-
-    page_iterator = paginator.paginate(
-        Bucket=bucket_name,
-        Prefix=subfolder
+def remove_csv_object(object_key):
+    obj = get_s3_object(
+        current_app.config['CSV_UPLOAD_BUCKET']['bucket'],
+        object_key,
+        current_app.config['CSV_UPLOAD_BUCKET']['access_key_id'],
+        current_app.config['CSV_UPLOAD_BUCKET']['secret_access_key'],
+        current_app.config['CSV_UPLOAD_BUCKET']['region']
     )
-
-    for page in page_iterator:
-        for obj in page.get('Contents', []):
-            key = obj['Key']
-            if key.lower().endswith(suffix.lower()):
-                if not last_modified or obj['LastModified'] >= last_modified:
-                    yield key
+    return obj.delete()

app/billing/billing_schemas.py

@@ -1,7 +1,7 @@
 from datetime import datetime
 
 create_or_update_free_sms_fragment_limit_schema = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
+    "$schema": "http://json-schema.org/draft-07/schema#",
     "description": "POST annual billing schema",
     "type": "object",
     "title": "Create",
@@ -12,30 +12,33 @@ create_or_update_free_sms_fragment_limit_schema = {
 }
 
 
-def serialize_ft_billing_remove_emails(data):
-    results = []
-    billed_notifications = [x for x in data if x.notification_type != 'email']
-    for notification in billed_notifications:
-        json_result = {
-            "month": (datetime.strftime(notification.month, "%B")),
-            "notification_type": notification.notification_type,
-            "billing_units": notification.billable_units,
-            "rate": float(notification.rate),
-            "postage": notification.postage,
+def serialize_ft_billing_remove_emails(rows):
+    return [
+        {
+            "month": (datetime.strftime(row.month, "%B")),
+            "notification_type": row.notification_type,
+            "chargeable_units": row.chargeable_units,
+            "notifications_sent": row.notifications_sent,
+            "rate": float(row.rate),
+            "cost": float(row.cost),
+            "free_allowance_used": row.free_allowance_used,
+            "charged_units": row.charged_units,
         }
-        results.append(json_result)
-    return results
+        for row in rows
+        if row.notification_type != 'email'
+    ]
 
 
-def serialize_ft_billing_yearly_totals(data):
-    yearly_totals = []
-    for total in data:
-        json_result = {
-            "notification_type": total.notification_type,
-            "billing_units": total.billable_units,
-            "rate": float(total.rate),
-            "letter_total": float(total.billable_units * total.rate) if total.notification_type == 'letter' else 0
+def serialize_ft_billing_yearly_totals(rows):
+    return [
+        {
+            "notification_type": row.notification_type,
+            "chargeable_units": row.chargeable_units,
+            "notifications_sent": row.notifications_sent,
+            "rate": float(row.rate),
+            "cost": float(row.cost),
+            "free_allowance_used": row.free_allowance_used,
+            "charged_units": row.charged_units,
         }
-        yearly_totals.append(json_result)
-
-    return yearly_totals
+        for row in rows
+    ]

app/billing/rest.py

@@ -11,7 +11,7 @@ from app.dao.annual_billing_dao import (
     dao_update_annual_billing_for_future_years,
     set_default_free_allowance_for_service,
 )
-from app.dao.date_util import get_current_financial_year_start_year
+from app.dao.date_util import get_current_calendar_year_start_year
 from app.dao.fact_billing_dao import (
     fetch_billing_totals_for_year,
     fetch_monthly_billing_for_year,
@@ -30,7 +30,6 @@ billing_blueprint = Blueprint(
 register_errors(billing_blueprint)
 
 
-@billing_blueprint.route('/ft-monthly-usage')
 @billing_blueprint.route('/monthly-usage')
 def get_yearly_usage_by_monthly_from_ft_billing(service_id):
     try:
@@ -42,7 +41,6 @@ def get_yearly_usage_by_monthly_from_ft_billing(service_id):
     return jsonify(data)
 
 
-@billing_blueprint.route('/ft-yearly-usage-summary')
 @billing_blueprint.route('/yearly-usage-summary')
 def get_yearly_billing_usage_summary_from_ft_billing(service_id):
     try:
@@ -65,7 +63,7 @@ def get_free_sms_fragment_limit(service_id):
     if annual_billing is None:
         service = Service.query.get(service_id)
         # An entry does not exist in annual_billing table for that service and year.
-        # Set the annual billing to the default free allowance based on the organisation type of the service.
+        # Set the annual billing to the default free allowance based on the organization type of the service.
 
         annual_billing = set_default_free_allowance_for_service(
             service=service,
@@ -89,7 +87,7 @@ def create_or_update_free_sms_fragment_limit(service_id):
 
 
 def update_free_sms_fragment_limit_data(service_id, free_sms_fragment_limit, financial_year_start):
-    current_year = get_current_financial_year_start_year()
+    current_year = get_current_calendar_year_start_year()
     if not financial_year_start:
         financial_year_start = current_year
 

app/broadcast_message/broadcast_message_schema.py

@@ -1,60 +0,0 @@
-from app.models import BroadcastStatusType
-from app.schema_validation.definitions import uuid
-
-create_broadcast_message_schema = {
-    '$schema': 'http://json-schema.org/draft-04/schema#',
-    'description': 'POST create broadcast_message schema',
-    'type': 'object',
-    'title': 'Create broadcast_message',
-    'properties': {
-        'template_id': uuid,
-        'service_id': uuid,
-        'created_by': uuid,
-        'personalisation': {'type': 'object'},
-        'starts_at': {'type': 'string', 'format': 'datetime'},
-        'finishes_at': {'type': 'string', 'format': 'datetime'},
-        'areas': {'type': 'object'},
-        'content': {'type': 'string', 'minLength': 1},
-        'reference': {'type': 'string', 'minLength': 1, 'maxLength': 255},
-    },
-    'required': ['service_id', 'created_by'],
-    'allOf': [
-        {'oneOf': [
-            {'required': ['template_id']},
-            {'required': ['content']},
-        ]},
-        {'oneOf': [
-            {'required': ['template_id']},
-            {'required': ['reference']},
-        ]},
-    ],
-    'additionalProperties': False
-}
-
-update_broadcast_message_schema = {
-    '$schema': 'http://json-schema.org/draft-04/schema#',
-    'description': 'POST update broadcast_message schema',
-    'type': 'object',
-    'title': 'Update broadcast_message',
-    'properties': {
-        'personalisation': {'type': 'object'},
-        'starts_at': {'type': 'string', 'format': 'datetime'},
-        'finishes_at': {'type': 'string', 'format': 'datetime'},
-        'areas': {'type': 'object'},
-    },
-    'required': [],
-    'additionalProperties': False
-}
-
-update_broadcast_message_status_schema = {
-    '$schema': 'http://json-schema.org/draft-04/schema#',
-    'description': 'POST update broadcast_message status schema',
-    'type': 'object',
-    'title': 'Update broadcast_message',
-    'properties': {
-        'status': {'type': 'string', 'enum': BroadcastStatusType.STATUSES},
-        'created_by': uuid,
-    },
-    'required': ['status', 'created_by'],
-    'additionalProperties': False
-}

app/broadcast_message/rest.py

@@ -1,165 +0,0 @@
-import iso8601
-from flask import Blueprint, jsonify, request
-from notifications_utils.template import BroadcastMessageTemplate
-
-from app.broadcast_message import utils as broadcast_utils
-from app.broadcast_message.broadcast_message_schema import (
-    create_broadcast_message_schema,
-    update_broadcast_message_schema,
-    update_broadcast_message_status_schema,
-)
-from app.dao.broadcast_message_dao import (
-    dao_get_broadcast_message_by_id_and_service_id,
-    dao_get_broadcast_messages_for_service,
-)
-from app.dao.dao_utils import dao_save_object
-from app.dao.services_dao import dao_fetch_service_by_id
-from app.dao.templates_dao import dao_get_template_by_id_and_service_id
-from app.dao.users_dao import get_user_by_id
-from app.errors import InvalidRequest, register_errors
-from app.models import BroadcastMessage, BroadcastStatusType
-from app.schema_validation import validate
-
-broadcast_message_blueprint = Blueprint(
-    'broadcast_message',
-    __name__,
-    url_prefix='/service/<uuid:service_id>/broadcast-message'
-)
-register_errors(broadcast_message_blueprint)
-
-
-def _parse_nullable_datetime(dt):
-    if dt:
-        return iso8601.parse_date(dt).replace(tzinfo=None)
-    return dt
-
-
-@broadcast_message_blueprint.route('', methods=['GET'])
-def get_broadcast_messages_for_service(service_id):
-    # TODO: should this return template content/data in some way? or can we rely on them being cached admin side.
-    # we might need stuff like template name for showing on the dashboard.
-    # TODO: should this paginate or filter on dates or anything?
-    broadcast_messages = [o.serialize() for o in dao_get_broadcast_messages_for_service(service_id)]
-    return jsonify(broadcast_messages=broadcast_messages)
-
-
-@broadcast_message_blueprint.route('/<uuid:broadcast_message_id>', methods=['GET'])
-def get_broadcast_message(service_id, broadcast_message_id):
-    return jsonify(dao_get_broadcast_message_by_id_and_service_id(broadcast_message_id, service_id).serialize())
-
-
-@broadcast_message_blueprint.route('', methods=['POST'])
-def create_broadcast_message(service_id):
-    data = request.get_json()
-
-    validate(data, create_broadcast_message_schema)
-    service = dao_fetch_service_by_id(data['service_id'])
-    user = get_user_by_id(data['created_by'])
-    personalisation = data.get('personalisation', {})
-    template_id = data.get('template_id')
-
-    if template_id:
-        template = dao_get_template_by_id_and_service_id(
-            template_id, data['service_id']
-        )
-        content = str(template._as_utils_template_with_personalisation(
-            personalisation
-        ))
-        reference = None
-    else:
-        temporary_template = BroadcastMessageTemplate.from_content(data['content'])
-        if temporary_template.content_too_long:
-            raise InvalidRequest(
-                (
-                    f'Content must be '
-                    f'{temporary_template.max_content_count:,.0f} '
-                    f'characters or fewer'
-                ) + (
-                    ' (because it could not be GSM7 encoded)'
-                    if temporary_template.non_gsm_characters else ''
-                ),
-                status_code=400,
-            )
-        template = None
-        content = str(temporary_template)
-        reference = data['reference']
-
-    broadcast_message = BroadcastMessage(
-        service_id=service.id,
-        template_id=template_id,
-        template_version=template.version if template else None,
-        personalisation=personalisation,
-        areas=data.get("areas", {}),
-        status=BroadcastStatusType.DRAFT,
-        starts_at=_parse_nullable_datetime(data.get('starts_at')),
-        finishes_at=_parse_nullable_datetime(data.get('finishes_at')),
-        created_by_id=user.id,
-        content=content,
-        reference=reference,
-        stubbed=service.restricted
-    )
-
-    dao_save_object(broadcast_message)
-
-    return jsonify(broadcast_message.serialize()), 201
-
-
-@broadcast_message_blueprint.route('/<uuid:broadcast_message_id>', methods=['POST'])
-def update_broadcast_message(service_id, broadcast_message_id):
-    data = request.get_json()
-    validate(data, update_broadcast_message_schema)
-
-    broadcast_message = dao_get_broadcast_message_by_id_and_service_id(broadcast_message_id, service_id)
-
-    if broadcast_message.status not in BroadcastStatusType.PRE_BROADCAST_STATUSES:
-        raise InvalidRequest(
-            f'Cannot update broadcast_message {broadcast_message.id} while it has status {broadcast_message.status}',
-            status_code=400
-        )
-
-    areas = data.get("areas", {})
-
-    if ('ids' in areas and 'simple_polygons' not in areas) or ('ids' not in areas and 'simple_polygons' in areas):
-        raise InvalidRequest(
-            f'Cannot update broadcast_message {broadcast_message.id}, area IDs or polygons are missing.',
-            status_code=400
-        )
-
-    if 'personalisation' in data:
-        broadcast_message.personalisation = data['personalisation']
-    if 'starts_at' in data:
-        broadcast_message.starts_at = _parse_nullable_datetime(data['starts_at'])
-    if 'finishes_at' in data:
-        broadcast_message.finishes_at = _parse_nullable_datetime(data['finishes_at'])
-    if 'ids' in areas and 'simple_polygons' in areas:
-        broadcast_message.areas = areas
-
-    dao_save_object(broadcast_message)
-
-    return jsonify(broadcast_message.serialize()), 200
-
-
-@broadcast_message_blueprint.route('/<uuid:broadcast_message_id>/status', methods=['POST'])
-def update_broadcast_message_status(service_id, broadcast_message_id):
-    data = request.get_json()
-
-    validate(data, update_broadcast_message_status_schema)
-    broadcast_message = dao_get_broadcast_message_by_id_and_service_id(broadcast_message_id, service_id)
-
-    if not broadcast_message.service.active:
-        raise InvalidRequest("Updating broadcast message is not allowed: service is inactive ", 403)
-
-    new_status = data['status']
-    updating_user = get_user_by_id(data['created_by'])
-
-    if updating_user not in broadcast_message.service.users:
-        #  we allow platform admins to cancel broadcasts, and we don't check user if request was done via API
-        if not (new_status == BroadcastStatusType.CANCELLED and updating_user.platform_admin):
-            raise InvalidRequest(
-                f'User {updating_user.id} cannot update broadcast_message {broadcast_message.id} from other service',
-                status_code=400
-            )
-
-    broadcast_utils.update_broadcast_message_status(broadcast_message, new_status, updating_user)
-
-    return jsonify(broadcast_message.serialize()), 200

app/broadcast_message/translators.py

@@ -1,37 +0,0 @@
-from bs4 import BeautifulSoup
-
-
-def cap_xml_to_dict(cap_xml):
-    # This function assumes that it’s being passed valid CAP XML
-    cap = BeautifulSoup(cap_xml, "xml")
-    return {
-        "msgType": cap.alert.msgType.text,
-        "reference": cap.alert.identifier.text,
-        "references": (
-            # references to previous events belonging to the same alert
-            cap.alert.references.text if cap.alert.references else None
-        ),
-        "cap_event": cap.alert.info.event.text,
-        "category": cap.alert.info.category.text,
-        "expires": cap.alert.info.expires.text,
-        "content": cap.alert.info.description.text,
-        "areas": [
-            {
-                "name": area.areaDesc.text,
-                "polygons": [
-                    cap_xml_polygon_to_list(polygon.text)
-                    for polygon in area.find_all('polygon')
-                ]
-            }
-            for area in cap.alert.info.find_all('area')
-        ]
-    }
-
-
-def cap_xml_polygon_to_list(polygon_string):
-    return [
-        [
-            float(coordinate) for coordinate in pair.split(',')
-        ]
-        for pair in polygon_string.strip().split(' ')
-    ]

app/broadcast_message/utils.py

@@ -1,140 +0,0 @@
-import inspect
-from datetime import datetime
-
-from flask import current_app
-from notifications_utils.clients.zendesk.zendesk_client import (
-    NotifySupportTicket,
-)
-
-from app import zendesk_client
-from app.celery.broadcast_message_tasks import send_broadcast_event
-from app.config import QueueNames
-from app.dao.dao_utils import dao_save_object
-from app.errors import InvalidRequest
-from app.models import (
-    BroadcastEvent,
-    BroadcastEventMessageType,
-    BroadcastStatusType,
-)
-
-
-def update_broadcast_message_status(broadcast_message, new_status, updating_user=None, api_key_id=None):
-    _validate_broadcast_update(broadcast_message, new_status, updating_user)
-
-    if new_status == BroadcastStatusType.BROADCASTING:
-        broadcast_message.approved_at = datetime.utcnow()
-        broadcast_message.approved_by = updating_user
-
-    if new_status == BroadcastStatusType.CANCELLED:
-        broadcast_message.cancelled_at = datetime.utcnow()
-        broadcast_message.cancelled_by = updating_user
-        broadcast_message.cancelled_by_api_key_id = api_key_id
-
-    current_app.logger.info(
-        f'broadcast_message {broadcast_message.id} moving from {broadcast_message.status} to {new_status}'
-    )
-    broadcast_message.status = new_status
-
-    dao_save_object(broadcast_message)
-    _create_p1_zendesk_alert(broadcast_message)
-
-    if new_status in {BroadcastStatusType.BROADCASTING, BroadcastStatusType.CANCELLED}:
-        _create_broadcast_event(broadcast_message)
-
-
-def _validate_broadcast_update(broadcast_message, new_status, updating_user):
-    if new_status not in BroadcastStatusType.ALLOWED_STATUS_TRANSITIONS[broadcast_message.status]:
-        raise InvalidRequest(
-            f'Cannot move broadcast_message {broadcast_message.id} from {broadcast_message.status} to {new_status}',
-            status_code=400
-        )
-
-    if new_status == BroadcastStatusType.BROADCASTING:
-        # training mode services can approve their own broadcasts
-        if updating_user == broadcast_message.created_by and not broadcast_message.service.restricted:
-            raise InvalidRequest(
-                f'User {updating_user.id} cannot approve their own broadcast_message {broadcast_message.id}',
-                status_code=400
-            )
-        elif len(broadcast_message.areas['simple_polygons']) == 0:
-            raise InvalidRequest(
-                f'broadcast_message {broadcast_message.id} has no selected areas and so cannot be broadcasted.',
-                status_code=400
-            )
-
-
-def _create_p1_zendesk_alert(broadcast_message):
-    if current_app.config['NOTIFY_ENVIRONMENT'] != 'live':
-        return
-
-    if broadcast_message.status != BroadcastStatusType.BROADCASTING:
-        return
-
-    message = inspect.cleandoc(f"""
-        Broadcast Sent
-
-        https://www.notifications.service.gov.uk/services/{broadcast_message.service_id}/current-alerts/{broadcast_message.id}
-
-        Sent on channel {broadcast_message.service.broadcast_channel} to {broadcast_message.areas["names"]}.
-
-        Content starts "{broadcast_message.content[:100]}".
-
-        Follow the runbook to check the broadcast went out OK:
-        https://docs.google.com/document/d/1J99yOlfp4nQz6et0w5oJVqi-KywtIXkxrEIyq_g2XUs/edit#heading=h.lzr9aq5b4wg
-    """)
-
-    ticket = NotifySupportTicket(
-        subject='Live broadcast sent',
-        message=message,
-        ticket_type=NotifySupportTicket.TYPE_INCIDENT,
-        technical_ticket=True,
-        org_id=current_app.config['BROADCAST_ORGANISATION_ID'],
-        org_type='central',
-        service_id=str(broadcast_message.service_id),
-        p1=True
-    )
-    zendesk_client.send_ticket_to_zendesk(ticket)
-
-
-def _create_broadcast_event(broadcast_message):
-    """
-    If the service is live and the broadcast message is not stubbed, creates a broadcast event, stores it in the
-    database, and triggers the task to send the CAP XML off.
-    """
-    service = broadcast_message.service
-
-    if not broadcast_message.stubbed and not service.restricted:
-        msg_types = {
-            BroadcastStatusType.BROADCASTING: BroadcastEventMessageType.ALERT,
-            BroadcastStatusType.CANCELLED: BroadcastEventMessageType.CANCEL,
-        }
-
-        event = BroadcastEvent(
-            service=service,
-            broadcast_message=broadcast_message,
-            message_type=msg_types[broadcast_message.status],
-            transmitted_content={"body": broadcast_message.content},
-            transmitted_areas=broadcast_message.areas,
-            # TODO: Probably move this somewhere more standalone too and imply that it shouldn't change. Should it
-            # include a service based identifier too? eg "flood-warnings@notifications.service.gov.uk" or similar
-            transmitted_sender='notifications.service.gov.uk',
-
-            # TODO: Should this be set to now? Or the original starts_at?
-            transmitted_starts_at=broadcast_message.starts_at,
-            transmitted_finishes_at=broadcast_message.finishes_at,
-        )
-
-        dao_save_object(event)
-
-        send_broadcast_event.apply_async(
-            kwargs={'broadcast_event_id': str(event.id)},
-            queue=QueueNames.BROADCASTS
-        )
-    elif broadcast_message.stubbed != service.restricted:
-        # It's possible for a service to create a broadcast in trial mode, and then approve it after the
-        # service is live (or vice versa). We don't think it's safe to send such broadcasts, as the service
-        # has changed since they were created. Log an error instead.
-        current_app.logger.error(
-            f'Broadcast event not created. Stubbed status of broadcast message was {broadcast_message.stubbed}'
-            f' but service was {"in trial mode" if service.restricted else "live"}'
-        )

app/celery/broadcast_message_tasks.py

@@ -1,230 +0,0 @@
-from datetime import datetime
-
-from flask import current_app
-
-from app import cbc_proxy_client, notify_celery
-from app.clients.cbc_proxy import CBCProxyRetryableException
-from app.config import QueueNames, TaskNames
-from app.dao.broadcast_message_dao import (
-    create_broadcast_provider_message,
-    dao_get_broadcast_event_by_id,
-    update_broadcast_provider_message_status,
-)
-from app.models import (
-    BroadcastEventMessageType,
-    BroadcastProvider,
-    BroadcastProviderMessageStatus,
-)
-from app.utils import format_sequential_number
-
-
-class BroadcastIntegrityError(Exception):
-    pass
-
-
-def get_retry_delay(retry_count):
-    """
-    Given a count of retries so far, return a delay for the next one.
-    `retry_count` should be 0 the first time a task fails.
-    """
-    # TODO: replace with celery's built in exponential backoff
-
-    # 2 to the power of x. 1, 2, 4, 8, 16, 32, ...
-    delay = 2**retry_count
-    # never wait longer than 4 minutes
-    return min(delay, 240)
-
-
-def check_event_is_authorised_to_be_sent(broadcast_event, provider):
-    if not broadcast_event.service.active:
-        raise BroadcastIntegrityError(
-            f'Cannot send broadcast_event {broadcast_event.id} ' +
-            f'to provider {provider}: the service is suspended'
-        )
-
-    if broadcast_event.service.restricted:
-        raise BroadcastIntegrityError(
-            f'Cannot send broadcast_event {broadcast_event.id} ' +
-            f'to provider {provider}: the service is not live'
-        )
-
-    if broadcast_event.broadcast_message.stubbed:
-        raise BroadcastIntegrityError(
-            f'Cannot send broadcast_event {broadcast_event.id} ' +
-            f'to provider {provider}: the broadcast message is stubbed'
-        )
-
-
-def check_event_makes_sense_in_sequence(broadcast_event, provider):
-    """
-    If any previous event hasn't sent yet for that provider, then we shouldn't send the current event. Instead, fail and
-    raise a zendesk ticket - so that a notify team member can assess the state of the previous messages, and if
-    necessary, can replay the `send_broadcast_provider_message` task if the previous message has now been sent.
-
-    Note: This is called before the new broadcast_provider_message is created.
-
-    # Help, I've come across this code following a pagerduty alert, what should I do?
-
-    1. Find the failing broadcast_provider_message associated with the previous event that caused this to trip.
-    2. If that provider message is still failing to send, fix the issue causing that. The task to send that previous
-       message might still be retrying in the background - look for logs related to that task.
-    3. If that provider message has sent succesfully, you might need to send this task off depending on context. This
-       might not always be true though, for example, it may not be necessary to send a cancel if the original alert has
-       already expired.
-    4. If you need to re-send this task off again, you'll need to run the following command on paas:
-       `send_broadcast_provider_message.apply_async(args=(broadcast_event_id, provider), queue=QueueNames.BROADCASTS)`
-    """
-    current_provider_message = broadcast_event.get_provider_message(provider)
-    # if this is the first time a task is being executed, it won't have a provider message yet
-    if current_provider_message and current_provider_message.status != BroadcastProviderMessageStatus.SENDING:
-        raise BroadcastIntegrityError(
-            f'Cannot send broadcast_event {broadcast_event.id} ' +
-            f'to provider {provider}: ' +
-            f'It is in status {current_provider_message.status}'
-        )
-
-    if broadcast_event.transmitted_finishes_at < datetime.utcnow():
-        raise BroadcastIntegrityError(
-            f'Cannot send broadcast_event {broadcast_event.id} ' +
-            f'to provider {provider}: ' +
-            f'The expiry time of {broadcast_event.transmitted_finishes_at} has already passed'
-        )
-
-    # get events sorted from earliest to latest
-    events = sorted(broadcast_event.broadcast_message.events, key=lambda x: x.sent_at)
-
-    for prev_event in events:
-        if prev_event.id != broadcast_event.id and prev_event.sent_at < broadcast_event.sent_at:
-            # get the record from when that event was sent to the same provider
-            prev_provider_message = prev_event.get_provider_message(provider)
-
-            # the previous message hasn't even got round to running `send_broadcast_provider_message` yet.
-            if not prev_provider_message:
-                raise BroadcastIntegrityError(
-                    f'Cannot send {broadcast_event.id}. Previous event {prev_event.id} ' +
-                    f'(type {prev_event.message_type}) has no provider_message for provider {provider} yet.\n' +
-                    'You must ensure that the other event sends succesfully, then manually kick off this event ' +
-                    'again by re-running send_broadcast_provider_message for this event and provider.'
-                )
-
-            # if there's a previous message that has started but not finished sending (whether it fatally errored or is
-            # currently retrying)
-            if prev_provider_message.status != BroadcastProviderMessageStatus.ACK:
-                raise BroadcastIntegrityError(
-                    f'Cannot send {broadcast_event.id}. Previous event {prev_event.id} ' +
-                    f'(type {prev_event.message_type}) has not finished sending to provider {provider} yet.\n' +
-                    f'It is currently in status "{prev_provider_message.status}".\n' +
-                    'You must ensure that the other event sends succesfully, then manually kick off this event ' +
-                    'again by re-running send_broadcast_provider_message for this event and provider.'
-                )
-
-
-@notify_celery.task(name="send-broadcast-event")
-def send_broadcast_event(broadcast_event_id):
-    broadcast_event = dao_get_broadcast_event_by_id(broadcast_event_id)
-
-    notify_celery.send_task(
-        name=TaskNames.PUBLISH_GOVUK_ALERTS,
-        queue=QueueNames.GOVUK_ALERTS
-    )
-
-    for provider in broadcast_event.service.get_available_broadcast_providers():
-        send_broadcast_provider_message.apply_async(
-            kwargs={'broadcast_event_id': broadcast_event_id, 'provider': provider},
-            queue=QueueNames.BROADCASTS
-        )
-
-
-# max_retries=None: retry forever
-@notify_celery.task(bind=True, name="send-broadcast-provider-message", max_retries=None)
-def send_broadcast_provider_message(self, broadcast_event_id, provider):
-    if not current_app.config['CBC_PROXY_ENABLED']:
-        current_app.logger.info(
-            "CBC Proxy disabled, not sending broadcast_provider_message for "
-            f"broadcast_event_id {broadcast_event_id} with provider {provider}"
-        )
-        return
-
-    broadcast_event = dao_get_broadcast_event_by_id(broadcast_event_id)
-
-    check_event_is_authorised_to_be_sent(broadcast_event, provider)
-    check_event_makes_sense_in_sequence(broadcast_event, provider)
-
-    # the broadcast_provider_message may already exist if we retried previously
-    broadcast_provider_message = broadcast_event.get_provider_message(provider)
-    if broadcast_provider_message is None:
-        broadcast_provider_message = create_broadcast_provider_message(broadcast_event, provider)
-
-    formatted_message_number = None
-    if provider == BroadcastProvider.VODAFONE:
-        formatted_message_number = format_sequential_number(broadcast_provider_message.message_number)
-
-    current_app.logger.info(
-        f'Invoking cbc proxy to send broadcast_provider_message with ID of {broadcast_provider_message.id} '
-        f'and broadcast_event ID of {broadcast_event_id} '
-        f'msgType {broadcast_event.message_type}'
-    )
-
-    areas = [
-        {"polygon": polygon}
-        for polygon in broadcast_event.transmitted_areas["simple_polygons"]
-    ]
-
-    cbc_proxy_provider_client = cbc_proxy_client.get_proxy(provider)
-
-    try:
-        if broadcast_event.message_type == BroadcastEventMessageType.ALERT:
-            cbc_proxy_provider_client.create_and_send_broadcast(
-                identifier=str(broadcast_provider_message.id),
-                message_number=formatted_message_number,
-                headline="GOV.UK Notify Broadcast",
-                description=broadcast_event.transmitted_content['body'],
-                areas=areas,
-                sent=broadcast_event.sent_at_as_cap_datetime_string,
-                expires=broadcast_event.transmitted_finishes_at_as_cap_datetime_string,
-                channel=broadcast_event.service.broadcast_channel
-            )
-        elif broadcast_event.message_type == BroadcastEventMessageType.UPDATE:
-            cbc_proxy_provider_client.update_and_send_broadcast(
-                identifier=str(broadcast_provider_message.id),
-                message_number=formatted_message_number,
-                headline="GOV.UK Notify Broadcast",
-                description=broadcast_event.transmitted_content['body'],
-                areas=areas,
-                previous_provider_messages=broadcast_event.get_earlier_provider_messages(provider),
-                sent=broadcast_event.sent_at_as_cap_datetime_string,
-                expires=broadcast_event.transmitted_finishes_at_as_cap_datetime_string,
-                # We think an alert update should always go out on the same channel that created the alert
-                # We recognise there is a small risk with this code here that if the services channel was
-                # changed between an alert being sent out and then updated, then something might go wrong
-                # but we are relying on service channels changing almost never, and not mid incident
-                # We may consider in the future, changing this such that we store the channel a broadcast was
-                # sent on on the broadcast message itself and pick the value from there instead of the service
-                channel=broadcast_event.service.broadcast_channel
-            )
-        elif broadcast_event.message_type == BroadcastEventMessageType.CANCEL:
-            cbc_proxy_provider_client.cancel_broadcast(
-                identifier=str(broadcast_provider_message.id),
-                message_number=formatted_message_number,
-                previous_provider_messages=broadcast_event.get_earlier_provider_messages(provider),
-                sent=broadcast_event.sent_at_as_cap_datetime_string,
-            )
-    except CBCProxyRetryableException as exc:
-        delay = get_retry_delay(self.request.retries)
-        current_app.logger.exception(
-            f'Retrying send_broadcast_provider_message for broadcast event {broadcast_event_id}, '
-            f'provider message {broadcast_provider_message.id}, provider {provider} in {delay} seconds'
-        )
-
-        self.retry(
-            exc=exc,
-            countdown=delay,
-            queue=QueueNames.BROADCASTS,
-        )
-
-    update_broadcast_provider_message_status(broadcast_provider_message, status=BroadcastProviderMessageStatus.ACK)
-
-
-@notify_celery.task(name='trigger-link-test')
-def trigger_link_test(provider):
-    cbc_proxy_client.get_proxy(provider).send_link_test()

app/celery/letters_pdf_tasks.py

@@ -1,570 +0,0 @@
-from base64 import urlsafe_b64encode
-from datetime import datetime, timedelta
-from hashlib import sha512
-
-from botocore.exceptions import ClientError as BotoClientError
-from flask import current_app
-from notifications_utils.letter_timings import LETTER_PROCESSING_DEADLINE
-from notifications_utils.postal_address import PostalAddress
-from notifications_utils.timezones import convert_utc_to_bst
-
-from app import encryption, notify_celery
-from app.aws import s3
-from app.config import QueueNames, TaskNames
-from app.cronitor import cronitor
-from app.dao.notifications_dao import (
-    dao_get_letters_and_sheets_volume_by_postage,
-    dao_get_letters_to_be_printed,
-    dao_get_notification_by_reference,
-    dao_update_notification,
-    dao_update_notifications_by_reference,
-    get_notification_by_id,
-    update_notification_status_by_id,
-)
-from app.dao.templates_dao import dao_get_template_by_id
-from app.errors import VirusScanError
-from app.exceptions import NotificationTechnicalFailureException
-from app.letters.utils import (
-    LetterPDFNotFound,
-    ScanErrorType,
-    find_letter_pdf_in_s3,
-    generate_letter_pdf_filename,
-    get_billable_units_for_letter_page_count,
-    get_file_names_from_error_bucket,
-    get_folder_name,
-    get_reference_from_filename,
-    move_error_pdf_to_scan_bucket,
-    move_failed_pdf,
-    move_sanitised_letter_to_test_or_live_pdf_bucket,
-    move_scan_to_invalid_pdf_bucket,
-)
-from app.models import (
-    INTERNATIONAL_LETTERS,
-    INTERNATIONAL_POSTAGE_TYPES,
-    KEY_TYPE_NORMAL,
-    KEY_TYPE_TEST,
-    NOTIFICATION_CREATED,
-    NOTIFICATION_DELIVERED,
-    NOTIFICATION_PENDING_VIRUS_CHECK,
-    NOTIFICATION_TECHNICAL_FAILURE,
-    NOTIFICATION_VALIDATION_FAILED,
-    NOTIFICATION_VIRUS_SCAN_FAILED,
-    POSTAGE_TYPES,
-    RESOLVE_POSTAGE_FOR_FILE_NAME,
-    Service,
-)
-
-
-@notify_celery.task(bind=True, name="get-pdf-for-templated-letter", max_retries=15, default_retry_delay=300)
-def get_pdf_for_templated_letter(self, notification_id):
-    try:
-        notification = get_notification_by_id(notification_id, _raise=True)
-        letter_filename = generate_letter_pdf_filename(
-            reference=notification.reference,
-            created_at=notification.created_at,
-            ignore_folder=notification.key_type == KEY_TYPE_TEST,
-            postage=notification.postage
-        )
-        letter_data = {
-            'letter_contact_block': notification.reply_to_text,
-            'template': {
-                "subject": notification.template.subject,
-                "content": notification.template.content,
-                "template_type": notification.template.template_type
-            },
-            'values': notification.personalisation,
-            'logo_filename': notification.service.letter_branding and notification.service.letter_branding.filename,
-            'letter_filename': letter_filename,
-            "notification_id": str(notification_id),
-            'key_type': notification.key_type
-        }
-
-        encrypted_data = encryption.encrypt(letter_data)
-
-        notify_celery.send_task(
-            name=TaskNames.CREATE_PDF_FOR_TEMPLATED_LETTER,
-            args=(encrypted_data,),
-            queue=QueueNames.SANITISE_LETTERS
-        )
-    except Exception as e:
-        try:
-            current_app.logger.exception(
-                f"RETRY: calling create-letter-pdf task for notification {notification_id} failed"
-            )
-            self.retry(exc=e, queue=QueueNames.RETRY)
-        except self.MaxRetriesExceededError:
-            message = f"RETRY FAILED: Max retries reached. " \
-                      f"The task create-letter-pdf failed for notification id {notification_id}. " \
-                      f"Notification has been updated to technical-failure"
-            update_notification_status_by_id(notification_id, NOTIFICATION_TECHNICAL_FAILURE)
-            raise NotificationTechnicalFailureException(message)
-
-
-@notify_celery.task(bind=True, name="update-billable-units-for-letter", max_retries=15, default_retry_delay=300)
-def update_billable_units_for_letter(self, notification_id, page_count):
-    notification = get_notification_by_id(notification_id, _raise=True)
-
-    billable_units = get_billable_units_for_letter_page_count(page_count)
-
-    if notification.key_type != KEY_TYPE_TEST:
-        notification.billable_units = billable_units
-        dao_update_notification(notification)
-
-        current_app.logger.info(
-            f"Letter notification id: {notification_id} reference {notification.reference}: "
-            f"billable units set to {billable_units}"
-        )
-
-
-@notify_celery.task(
-    bind=True, name="update-validation-failed-for-templated-letter", max_retries=15, default_retry_delay=300
-)
-def update_validation_failed_for_templated_letter(self, notification_id, page_count):
-    notification = get_notification_by_id(notification_id, _raise=True)
-    notification.status = NOTIFICATION_VALIDATION_FAILED
-    dao_update_notification(notification)
-    current_app.logger.info(f"Validation failed: letter is too long {page_count} for letter with id: {notification_id}")
-
-
-@notify_celery.task(name='collate-letter-pdfs-to-be-sent')
-@cronitor("collate-letter-pdfs-to-be-sent")
-def collate_letter_pdfs_to_be_sent():
-    """
-    Finds all letters which are still waiting to be sent to DVLA for printing
-
-    This would usually be run at 5.50pm and collect up letters created between before 5:30pm today
-    that have not yet been sent.
-    If run after midnight, it will collect up letters created before 5:30pm the day before.
-    """
-    print_run_date = convert_utc_to_bst(datetime.utcnow())
-    if print_run_date.time() < LETTER_PROCESSING_DEADLINE:
-        print_run_date = print_run_date - timedelta(days=1)
-
-    print_run_deadline = print_run_date.replace(
-        hour=17, minute=30, second=0, microsecond=0
-    )
-    _get_letters_and_sheets_volumes_and_send_to_dvla(print_run_deadline)
-
-    for postage in POSTAGE_TYPES:
-        current_app.logger.info(f"starting collate-letter-pdfs-to-be-sent processing for postage class {postage}")
-        letters_to_print = get_key_and_size_of_letters_to_be_sent_to_print(print_run_deadline, postage)
-
-        for i, letters in enumerate(group_letters(letters_to_print)):
-            filenames = [letter['Key'] for letter in letters]
-
-            service_id = letters[0]['ServiceId']
-            organisation_id = letters[0]['OrganisationId']
-
-            hash = urlsafe_b64encode(sha512(''.join(filenames).encode()).digest())[:20].decode()
-            # eg NOTIFY.2018-12-31.001.Wjrui5nAvObjPd-3GEL-.ZIP
-            dvla_filename = 'NOTIFY.{date}.{postage}.{num:03}.{hash}.{service_id}.{organisation_id}.ZIP'.format(
-                date=print_run_deadline.strftime("%Y-%m-%d"),
-                postage=RESOLVE_POSTAGE_FOR_FILE_NAME[postage],
-                num=i + 1,
-                hash=hash,
-                service_id=service_id,
-                organisation_id=organisation_id
-            )
-
-            current_app.logger.info(
-                'Calling task zip-and-send-letter-pdfs for {} pdfs to upload {} with total size {:,} bytes'.format(
-                    len(filenames),
-                    dvla_filename,
-                    sum(letter['Size'] for letter in letters)
-                )
-            )
-            notify_celery.send_task(
-                name=TaskNames.ZIP_AND_SEND_LETTER_PDFS,
-                kwargs={
-                    'filenames_to_zip': filenames,
-                    'upload_filename': dvla_filename
-                },
-                queue=QueueNames.PROCESS_FTP,
-                compression='zlib'
-            )
-        current_app.logger.info(f"finished collate-letter-pdfs-to-be-sent processing for postage class {postage}")
-
-    current_app.logger.info("finished collate-letter-pdfs-to-be-sent")
-
-
-def _get_letters_and_sheets_volumes_and_send_to_dvla(print_run_deadline):
-    letters_volumes = dao_get_letters_and_sheets_volume_by_postage(print_run_deadline)
-    send_letters_volume_email_to_dvla(letters_volumes, print_run_deadline.date())
-
-
-def send_letters_volume_email_to_dvla(letters_volumes, date):
-    personalisation = {
-        'total_volume': 0,
-        'first_class_volume': 0,
-        'second_class_volume': 0,
-        'international_volume': 0,
-        'total_sheets': 0,
-        'first_class_sheets': 0,
-        "second_class_sheets": 0,
-        'international_sheets': 0,
-        'date': date.strftime("%d %B %Y")
-    }
-    for item in letters_volumes:
-        personalisation['total_volume'] += item.letters_count
-        personalisation['total_sheets'] += item.sheets_count
-        if f"{item.postage}_class_volume" in personalisation:
-            personalisation[f"{item.postage}_class_volume"] = item.letters_count
-            personalisation[f"{item.postage}_class_sheets"] = item.sheets_count
-        else:
-            personalisation["international_volume"] += item.letters_count
-            personalisation["international_sheets"] += item.sheets_count
-
-    template = dao_get_template_by_id(current_app.config['LETTERS_VOLUME_EMAIL_TEMPLATE_ID'])
-    recipients = current_app.config['DVLA_EMAIL_ADDRESSES']
-    reply_to = template.service.get_default_reply_to_email_address()
-    service = Service.query.get(current_app.config['NOTIFY_SERVICE_ID'])
-
-    # avoid circular imports:
-    from app.notifications.process_notifications import (
-        persist_notification,
-        send_notification_to_queue,
-    )
-    for recipient in recipients:
-        saved_notification = persist_notification(
-            template_id=template.id,
-            template_version=template.version,
-            recipient=recipient,
-            service=service,
-            personalisation=personalisation,
-            notification_type=template.template_type,
-            api_key_id=None,
-            key_type=KEY_TYPE_NORMAL,
-            reply_to_text=reply_to
-        )
-
-        send_notification_to_queue(saved_notification, False, queue=QueueNames.NOTIFY)
-
-
-def get_key_and_size_of_letters_to_be_sent_to_print(print_run_deadline, postage):
-    letters_awaiting_sending = dao_get_letters_to_be_printed(print_run_deadline, postage)
-    for letter in letters_awaiting_sending:
-        try:
-            letter_pdf = find_letter_pdf_in_s3(letter)
-            yield {
-                "Key": letter_pdf.key,
-                "Size": letter_pdf.size,
-                "ServiceId": str(letter.service_id),
-                "OrganisationId": str(letter.service.organisation_id)
-            }
-        except (BotoClientError, LetterPDFNotFound) as e:
-            current_app.logger.exception(
-                f"Error getting letter from bucket for notification: {letter.id} with reference: {letter.reference}", e)
-
-
-def group_letters(letter_pdfs):
-    """
-    Group letters in chunks of MAX_LETTER_PDF_ZIP_FILESIZE. Will add files to lists, never going over that size.
-    If a single file is (somehow) larger than MAX_LETTER_PDF_ZIP_FILESIZE that'll be in a list on it's own.
-    If there are no files, will just exit (rather than yielding an empty list).
-    """
-    running_filesize = 0
-    list_of_files = []
-    service_id = None
-    for letter in letter_pdfs:
-        if letter['Key'].lower().endswith('.pdf'):
-            if not service_id:
-                service_id = letter['ServiceId']
-            if (
-                running_filesize + letter['Size'] > current_app.config['MAX_LETTER_PDF_ZIP_FILESIZE']
-                or len(list_of_files) >= current_app.config['MAX_LETTER_PDF_COUNT_PER_ZIP']
-                or letter['ServiceId'] != service_id
-            ):
-                yield list_of_files
-                running_filesize = 0
-                list_of_files = []
-                service_id = None
-
-            if not service_id:
-                service_id = letter['ServiceId']
-            running_filesize += letter['Size']
-            list_of_files.append(letter)
-
-    if list_of_files:
-        yield list_of_files
-
-
-@notify_celery.task(bind=True, name='sanitise-letter', max_retries=15, default_retry_delay=300)
-def sanitise_letter(self, filename):
-    try:
-        reference = get_reference_from_filename(filename)
-        notification = dao_get_notification_by_reference(reference)
-
-        current_app.logger.info('Notification ID {} Virus scan passed: {}'.format(notification.id, filename))
-
-        if notification.status != NOTIFICATION_PENDING_VIRUS_CHECK:
-            current_app.logger.info('Sanitise letter called for notification {} which is in {} state'.format(
-                notification.id, notification.status))
-            return
-
-        notify_celery.send_task(
-            name=TaskNames.SANITISE_LETTER,
-            kwargs={
-                'notification_id': str(notification.id),
-                'filename': filename,
-                'allow_international_letters': notification.service.has_permission(
-                    INTERNATIONAL_LETTERS
-                ),
-            },
-            queue=QueueNames.SANITISE_LETTERS,
-        )
-    except Exception:
-        try:
-            current_app.logger.exception(
-                "RETRY: calling sanitise_letter task for notification {} failed".format(notification.id)
-            )
-            self.retry(queue=QueueNames.RETRY)
-        except self.MaxRetriesExceededError:
-            message = "RETRY FAILED: Max retries reached. " \
-                      "The task sanitise_letter failed for notification {}. " \
-                      "Notification has been updated to technical-failure".format(notification.id)
-            update_notification_status_by_id(notification.id, NOTIFICATION_TECHNICAL_FAILURE)
-            raise NotificationTechnicalFailureException(message)
-
-
-@notify_celery.task(bind=True, name='process-sanitised-letter', max_retries=15, default_retry_delay=300)
-def process_sanitised_letter(self, sanitise_data):
-    letter_details = encryption.decrypt(sanitise_data)
-
-    filename = letter_details['filename']
-    notification_id = letter_details['notification_id']
-
-    current_app.logger.info('Processing sanitised letter with id {}'.format(notification_id))
-    notification = get_notification_by_id(notification_id, _raise=True)
-
-    if notification.status != NOTIFICATION_PENDING_VIRUS_CHECK:
-        current_app.logger.info(
-            'process-sanitised-letter task called for notification {} which is in {} state'.format(
-                notification.id, notification.status)
-        )
-        return
-
-    try:
-        original_pdf_object = s3.get_s3_object(current_app.config['LETTERS_SCAN_BUCKET_NAME'], filename)
-
-        if letter_details['validation_status'] == 'failed':
-            current_app.logger.info('Processing invalid precompiled pdf with id {} (file {})'.format(
-                notification_id, filename))
-
-            _move_invalid_letter_and_update_status(
-                notification=notification,
-                filename=filename,
-                scan_pdf_object=original_pdf_object,
-                message=letter_details['message'],
-                invalid_pages=letter_details['invalid_pages'],
-                page_count=letter_details['page_count'],
-            )
-            return
-
-        current_app.logger.info('Processing valid precompiled pdf with id {} (file {})'.format(
-            notification_id, filename))
-
-        billable_units = get_billable_units_for_letter_page_count(letter_details['page_count'])
-        is_test_key = notification.key_type == KEY_TYPE_TEST
-
-        # Updating the notification needs to happen before the file is moved. This is so that if updating the
-        # notification fails, the task can retry because the file is in the same place.
-        update_letter_pdf_status(
-            reference=notification.reference,
-            status=NOTIFICATION_DELIVERED if is_test_key else NOTIFICATION_CREATED,
-            billable_units=billable_units,
-            recipient_address=letter_details['address']
-        )
-
-        # The original filename could be wrong because we didn't know the postage.
-        # Now we know if the letter is international, we can check what the filename should be.
-        upload_file_name = generate_letter_pdf_filename(
-            reference=notification.reference,
-            created_at=notification.created_at,
-            ignore_folder=True,
-            postage=notification.postage
-        )
-
-        move_sanitised_letter_to_test_or_live_pdf_bucket(
-            filename,
-            is_test_key,
-            notification.created_at,
-            upload_file_name,
-        )
-        # We've moved the sanitised PDF from the sanitise bucket, but still need to delete the original file:
-        original_pdf_object.delete()
-
-    except BotoClientError:
-        # Boto exceptions are likely to be caused by the file(s) being in the wrong place, so retrying won't help -
-        # we'll need to manually investigate
-        current_app.logger.exception(
-            f"Boto error when processing sanitised letter for notification {notification.id} (file {filename})"
-        )
-        update_notification_status_by_id(notification.id, NOTIFICATION_TECHNICAL_FAILURE)
-        raise NotificationTechnicalFailureException
-    except Exception:
-        try:
-            current_app.logger.exception(
-                "RETRY: calling process_sanitised_letter task for notification {} failed".format(notification.id)
-            )
-            self.retry(queue=QueueNames.RETRY)
-        except self.MaxRetriesExceededError:
-            message = "RETRY FAILED: Max retries reached. " \
-                      "The task process_sanitised_letter failed for notification {}. " \
-                      "Notification has been updated to technical-failure".format(notification.id)
-            update_notification_status_by_id(notification.id, NOTIFICATION_TECHNICAL_FAILURE)
-            raise NotificationTechnicalFailureException(message)
-
-
-def _move_invalid_letter_and_update_status(
-    *, notification, filename, scan_pdf_object, message=None, invalid_pages=None, page_count=None
-):
-    try:
-        move_scan_to_invalid_pdf_bucket(
-            source_filename=filename,
-            message=message,
-            invalid_pages=invalid_pages,
-            page_count=page_count
-        )
-        scan_pdf_object.delete()
-
-        update_letter_pdf_status(
-            reference=notification.reference,
-            status=NOTIFICATION_VALIDATION_FAILED,
-            billable_units=0)
-    except BotoClientError:
-        current_app.logger.exception(
-            "Error when moving letter with id {} to invalid PDF bucket".format(notification.id)
-        )
-        update_notification_status_by_id(notification.id, NOTIFICATION_TECHNICAL_FAILURE)
-        raise NotificationTechnicalFailureException
-
-
-@notify_celery.task(name='process-virus-scan-failed')
-def process_virus_scan_failed(filename):
-    move_failed_pdf(filename, ScanErrorType.FAILURE)
-    reference = get_reference_from_filename(filename)
-    notification = dao_get_notification_by_reference(reference)
-    updated_count = update_letter_pdf_status(reference, NOTIFICATION_VIRUS_SCAN_FAILED, billable_units=0)
-
-    if updated_count != 1:
-        raise Exception(
-            "There should only be one letter notification for each reference. Found {} notifications".format(
-                updated_count
-            )
-        )
-
-    error = VirusScanError('notification id {} Virus scan failed: {}'.format(notification.id, filename))
-    current_app.logger.exception(error)
-    raise error
-
-
-@notify_celery.task(name='process-virus-scan-error')
-def process_virus_scan_error(filename):
-    move_failed_pdf(filename, ScanErrorType.ERROR)
-    reference = get_reference_from_filename(filename)
-    notification = dao_get_notification_by_reference(reference)
-    updated_count = update_letter_pdf_status(reference, NOTIFICATION_TECHNICAL_FAILURE, billable_units=0)
-
-    if updated_count != 1:
-        raise Exception(
-            "There should only be one letter notification for each reference. Found {} notifications".format(
-                updated_count
-            )
-        )
-    error = VirusScanError('notification id {} Virus scan error: {}'.format(notification.id, filename))
-    current_app.logger.exception(error)
-    raise error
-
-
-def update_letter_pdf_status(reference, status, billable_units, recipient_address=None):
-    postage = None
-    if recipient_address:
-        # fix allow_international_letters
-        postage = PostalAddress(raw_address=recipient_address.replace(',', '\n'),
-                                allow_international_letters=True
-                                ).postage
-        postage = postage if postage in INTERNATIONAL_POSTAGE_TYPES else None
-    update_dict = {'status': status, 'billable_units': billable_units, 'updated_at': datetime.utcnow()}
-    if postage:
-        update_dict.update({'postage': postage, 'international': True})
-    if recipient_address:
-        update_dict['to'] = recipient_address
-        update_dict['normalised_to'] = ''.join(recipient_address.split()).lower()
-    return dao_update_notifications_by_reference(
-        references=[reference],
-        update_dict=update_dict)[0]
-
-
-def replay_letters_in_error(filename=None):
-    # This method can be used to replay letters that end up in the ERROR directory.
-    # We had an incident where clamAV was not processing the virus scan.
-    if filename:
-        move_error_pdf_to_scan_bucket(filename)
-        # call task to add the filename to anti virus queue
-        current_app.logger.info("Calling scan_file for: {}".format(filename))
-
-        if current_app.config['ANTIVIRUS_ENABLED']:
-            notify_celery.send_task(
-                name=TaskNames.SCAN_FILE,
-                kwargs={'filename': filename},
-                queue=QueueNames.ANTIVIRUS,
-            )
-        else:
-            # stub out antivirus in dev
-            sanitise_letter.apply_async(
-                [filename],
-                queue=QueueNames.LETTERS
-            )
-    else:
-        error_files = get_file_names_from_error_bucket()
-        for item in error_files:
-            moved_file_name = item.key.split('/')[1]
-            current_app.logger.info("Calling scan_file for: {}".format(moved_file_name))
-            move_error_pdf_to_scan_bucket(moved_file_name)
-            # call task to add the filename to anti virus queue
-            if current_app.config['ANTIVIRUS_ENABLED']:
-                notify_celery.send_task(
-                    name=TaskNames.SCAN_FILE,
-                    kwargs={'filename': moved_file_name},
-                    queue=QueueNames.ANTIVIRUS,
-                )
-            else:
-                # stub out antivirus in dev
-                sanitise_letter.apply_async(
-                    [filename],
-                    queue=QueueNames.LETTERS
-                )
-
-
-@notify_celery.task(name='resanitise-pdf')
-def resanitise_pdf(notification_id):
-    """
-    `notification_id` is the notification id for a PDF letter which was either uploaded or sent using the API.
-
-    This task calls the `recreate_pdf_for_precompiled_letter` template preview task which recreates the
-    PDF for a letter which is already sanitised and in the letters-pdf bucket. The new file that is generated
-    will then overwrite the existing letter in the letters-pdf bucket.
-    """
-    notification = get_notification_by_id(notification_id)
-
-    # folder_name is the folder that the letter is in the letters-pdf bucket e.g. '2021-10-10/'
-    folder_name = get_folder_name(notification.created_at)
-
-    filename = generate_letter_pdf_filename(
-            reference=notification.reference,
-            created_at=notification.created_at,
-            ignore_folder=True,
-            postage=notification.postage
-        )
-
-    notify_celery.send_task(
-        name=TaskNames.RECREATE_PDF_FOR_PRECOMPILED_LETTER,
-        kwargs={
-            'notification_id': str(notification.id),
-            'file_location': f'{folder_name}{filename}',
-            'allow_international_letters': notification.service.has_permission(
-                INTERNATIONAL_LETTERS
-            ),
-        },
-        queue=QueueNames.SANITISE_LETTERS,
-    )

app/celery/nightly_tasks.py

@@ -1,16 +1,12 @@
 from datetime import datetime, timedelta
 
-import pytz
 from flask import current_app
-from notifications_utils.clients.zendesk.zendesk_client import (
-    NotifySupportTicket,
-)
-from notifications_utils.timezones import convert_utc_to_bst
-from sqlalchemy import func
 from sqlalchemy.exc import SQLAlchemyError
 
-from app import notify_celery, statsd_client, zendesk_client
+from app import notify_celery
 from app.aws import s3
+from app.aws.s3 import remove_csv_object
+from app.celery.process_ses_receipts_tasks import check_and_queue_callback_task
 from app.config import QueueNames
 from app.cronitor import cronitor
 from app.dao.fact_processing_time_dao import insert_update_processing_time
@@ -18,6 +14,7 @@ from app.dao.inbound_sms_dao import delete_inbound_sms_older_than_retention
 from app.dao.jobs_dao import (
     dao_archive_job,
     dao_get_jobs_older_than_data_retention,
+    dao_get_unfinished_jobs,
 )
 from app.dao.notifications_dao import (
     dao_get_notifications_processing_time_stats,
@@ -28,19 +25,8 @@ from app.dao.notifications_dao import (
 from app.dao.service_data_retention_dao import (
     fetch_service_data_retention_for_all_services_by_notification_type,
 )
-from app.models import (
-    EMAIL_TYPE,
-    KEY_TYPE_NORMAL,
-    LETTER_TYPE,
-    NOTIFICATION_SENDING,
-    SMS_TYPE,
-    FactProcessingTime,
-    Notification,
-)
-from app.notifications.notifications_ses_callback import (
-    check_and_queue_callback_task,
-)
-from app.utils import get_london_midnight_in_utc
+from app.models import EMAIL_TYPE, SMS_TYPE, FactProcessingTime
+from app.utils import get_midnight_in_utc
 
 
 @notify_celery.task(name="remove_sms_email_jobs")
@@ -49,12 +35,6 @@ def remove_sms_email_csv_files():
     _remove_csv_files([EMAIL_TYPE, SMS_TYPE])
 
 
-@notify_celery.task(name="remove_letter_jobs")
-@cronitor("remove_letter_jobs")
-def remove_letter_csv_files():
-    _remove_csv_files([LETTER_TYPE])
-
-
 def _remove_csv_files(job_types):
     jobs = dao_get_jobs_older_than_data_retention(notification_types=job_types)
     for job in jobs:
@@ -63,11 +43,23 @@ def _remove_csv_files(job_types):
         current_app.logger.info("Job ID {} has been removed from s3.".format(job.id))
 
 
+@notify_celery.task(name="cleanup-unfinished-jobs")
+def cleanup_unfinished_jobs():
+    now = datetime.utcnow()
+    jobs = dao_get_unfinished_jobs()
+    for job in jobs:
+        # The query already checks that the processing_finished time is null, so here we are saying
+        # if it started more than 4 hours ago, that's too long
+        acceptable_finish_time = job.processing_started + timedelta(minutes=5)
+        if now > acceptable_finish_time:
+            remove_csv_object(job.original_file_name)
+            dao_archive_job(job)
+
+
 @notify_celery.task(name="delete-notifications-older-than-retention")
 def delete_notifications_older_than_retention():
     delete_email_notifications_older_than_retention.apply_async(queue=QueueNames.REPORTING)
     delete_sms_notifications_older_than_retention.apply_async(queue=QueueNames.REPORTING)
-    delete_letter_notifications_older_than_retention.apply_async(queue=QueueNames.REPORTING)
 
 
 @notify_celery.task(name="delete-sms-notifications")
@@ -82,19 +74,12 @@ def delete_email_notifications_older_than_retention():
     _delete_notifications_older_than_retention_by_type('email')
 
 
-@notify_celery.task(name="delete-letter-notifications")
-@cronitor("delete-letter-notifications")
-def delete_letter_notifications_older_than_retention():
-    _delete_notifications_older_than_retention_by_type('letter')
-
-
 def _delete_notifications_older_than_retention_by_type(notification_type):
     flexible_data_retention = fetch_service_data_retention_for_all_services_by_notification_type(notification_type)
 
     for f in flexible_data_retention:
-        day_to_delete_backwards_from = get_london_midnight_in_utc(
-            convert_utc_to_bst(datetime.utcnow()).date() - timedelta(days=f.days_of_retention)
-        )
+        day_to_delete_backwards_from = get_midnight_in_utc(datetime.utcnow()).date() \
+                                       - timedelta(days=f.days_of_retention)
 
         delete_notifications_for_service_and_type.apply_async(queue=QueueNames.REPORTING, kwargs={
             'service_id': f.service_id,
@@ -102,7 +87,8 @@ def _delete_notifications_older_than_retention_by_type(notification_type):
             'datetime_to_delete_before': day_to_delete_backwards_from
         })
 
-    seven_days_ago = get_london_midnight_in_utc(convert_utc_to_bst(datetime.utcnow()).date() - timedelta(days=7))
+    seven_days_ago = get_midnight_in_utc(datetime.utcnow()).date() - timedelta(days=7)
+
     service_ids_with_data_retention = {x.service_id for x in flexible_data_retention}
 
     # get a list of all service ids that we'll need to delete for. Typically that might only be 5% of services.
@@ -160,7 +146,6 @@ def timeout_notifications():
         notifications = dao_timeout_notifications(cutoff_time)
 
         for notification in notifications:
-            statsd_client.incr(f'timeout-sending.{notification.sent_by}')
             check_and_queue_callback_task(notification)
 
         current_app.logger.info(
@@ -185,126 +170,23 @@ def delete_inbound_sms():
         raise
 
 
-@notify_celery.task(name="raise-alert-if-letter-notifications-still-sending")
-@cronitor("raise-alert-if-letter-notifications-still-sending")
-def raise_alert_if_letter_notifications_still_sending():
-    still_sending_count, sent_date = get_letter_notifications_still_sending_when_they_shouldnt_be()
-
-    if still_sending_count:
-        message = "There are {} letters in the 'sending' state from {}".format(
-            still_sending_count,
-            sent_date.strftime('%A %d %B')
-        )
-        # Only send alerts in production
-        if current_app.config['NOTIFY_ENVIRONMENT'] in ['live', 'production', 'test']:
-            message += ". Resolve using https://github.com/alphagov/notifications-manuals/wiki/Support-Runbook#deal-with-letters-still-in-sending"  # noqa
-
-            ticket = NotifySupportTicket(
-                subject=f"[{current_app.config['NOTIFY_ENVIRONMENT']}] Letters still sending",
-                email_ccs=current_app.config['DVLA_EMAIL_ADDRESSES'],
-                message=message,
-                ticket_type=NotifySupportTicket.TYPE_INCIDENT,
-                technical_ticket=True,
-                ticket_categories=['notify_letters']
-            )
-            zendesk_client.send_ticket_to_zendesk(ticket)
-        else:
-            current_app.logger.info(message)
-
-
-def get_letter_notifications_still_sending_when_they_shouldnt_be():
-    today = datetime.utcnow().date()
-
-    # Do nothing on the weekend
-    if today.isoweekday() in {6, 7}:  # sat, sun
-        return 0, None
-
-    if today.isoweekday() in {1, 2}:  # mon, tues. look for files from before the weekend
-        offset_days = 4
-    else:
-        offset_days = 2
-
-    expected_sent_date = today - timedelta(days=offset_days)
-
-    q = Notification.query.filter(
-        Notification.notification_type == LETTER_TYPE,
-        Notification.status == NOTIFICATION_SENDING,
-        Notification.key_type == KEY_TYPE_NORMAL,
-        func.date(Notification.sent_at) <= expected_sent_date
-    )
-
-    return q.count(), expected_sent_date
-
-
-@notify_celery.task(name='raise-alert-if-no-letter-ack-file')
-@cronitor('raise-alert-if-no-letter-ack-file')
-def letter_raise_alert_if_no_ack_file_for_zip():
-    # get a list of zip files since yesterday
-    zip_file_set = set()
-    today_str = datetime.utcnow().strftime('%Y-%m-%d')
-    yesterday = datetime.now(tz=pytz.utc) - timedelta(days=1)  # AWS datetime format
-
-    for key in s3.get_list_of_files_by_suffix(bucket_name=current_app.config['LETTERS_PDF_BUCKET_NAME'],
-                                              subfolder=today_str + '/zips_sent',
-                                              suffix='.TXT'):
-        subname = key.split('/')[-1]  # strip subfolder in name
-        zip_file_set.add(subname.upper().replace('.ZIP.TXT', ''))
-
-    # get acknowledgement file
-    ack_file_set = set()
-
-    for key in s3.get_list_of_files_by_suffix(bucket_name=current_app.config['DVLA_RESPONSE_BUCKET_NAME'],
-                                              subfolder='root/dispatch', suffix='.ACK.txt', last_modified=yesterday):
-        ack_file_set.add(key.lstrip('root/dispatch').upper().replace('.ACK.TXT', ''))  # noqa
-
-    message = '\n'.join([
-        "Letter ack file does not contain all zip files sent."
-        "",
-        f"See runbook at https://github.com/alphagov/notifications-manuals/wiki/Support-Runbook#letter-ack-file-does-not-contain-all-zip-files-sent\n",  # noqa
-        f"pdf bucket: {current_app.config['LETTERS_PDF_BUCKET_NAME']}, subfolder: {datetime.utcnow().strftime('%Y-%m-%d')}/zips_sent",  # noqa
-        f"ack bucket: {current_app.config['DVLA_RESPONSE_BUCKET_NAME']}",
-        "",
-        f"Missing ack for zip files: {str(sorted(zip_file_set - ack_file_set))}",
-    ])
-
-    # strip empty element before comparison
-    ack_file_set.discard('')
-    zip_file_set.discard('')
-
-    if len(zip_file_set - ack_file_set) > 0:
-        if current_app.config['NOTIFY_ENVIRONMENT'] in ['live', 'production', 'test']:
-            ticket = NotifySupportTicket(
-                subject="Letter acknowledge error",
-                message=message,
-                ticket_type=NotifySupportTicket.TYPE_INCIDENT,
-                technical_ticket=True,
-                ticket_categories=['notify_letters']
-            )
-            zendesk_client.send_ticket_to_zendesk(ticket)
-        current_app.logger.error(message)
-
-    if len(ack_file_set - zip_file_set) > 0:
-        current_app.logger.info(
-            "letter ack contains zip that is not for today: {}".format(ack_file_set - zip_file_set)
-        )
-
-
 @notify_celery.task(name='save-daily-notification-processing-time')
 @cronitor("save-daily-notification-processing-time")
-def save_daily_notification_processing_time(bst_date=None):
-    # bst_date is a string in the format of "YYYY-MM-DD"
-    if bst_date is None:
-        # if a date is not provided, we run against yesterdays data
-        bst_date = (datetime.utcnow() - timedelta(days=1)).date()
-    else:
-        bst_date = datetime.strptime(bst_date, "%Y-%m-%d").date()
+def save_daily_notification_processing_time(local_date=None):
 
-    start_time = get_london_midnight_in_utc(bst_date)
-    end_time = get_london_midnight_in_utc(bst_date + timedelta(days=1))
+    # local_date is a string in the format of "YYYY-MM-DD"
+    if local_date is None:
+        # if a date is not provided, we run against yesterdays data
+        local_date = (datetime.utcnow() - timedelta(days=1)).date()
+    else:
+        local_date = datetime.strptime(local_date, "%Y-%m-%d").date()
+
+    start_time = get_midnight_in_utc(local_date)
+    end_time = get_midnight_in_utc(local_date + timedelta(days=1))
     result = dao_get_notifications_processing_time_stats(start_time, end_time)
     insert_update_processing_time(
         FactProcessingTime(
-            bst_date=bst_date,
+            local_date=local_date,
             messages_total=result.messages_total,
             messages_within_10_secs=result.messages_within_10_secs
         )

app/celery/process_ses_receipts_tasks.py

@@ -5,75 +5,91 @@ from celery.exceptions import Retry
 from flask import current_app, json
 from sqlalchemy.orm.exc import NoResultFound
 
-from app import notify_celery, statsd_client
-from app.clients.email.aws_ses import get_aws_responses
+from app import notify_celery
+from app.celery.service_callback_tasks import (
+    create_complaint_callback_data,
+    create_delivery_status_callback_data,
+    send_complaint_to_service,
+    send_delivery_status_to_service,
+)
 from app.config import QueueNames
 from app.dao import notifications_dao
-from app.models import NOTIFICATION_PENDING, NOTIFICATION_SENDING
-from app.notifications.notifications_ses_callback import (
-    _check_and_queue_complaint_callback_task,
-    check_and_queue_callback_task,
-    determine_notification_bounce_type,
-    handle_complaint,
+from app.dao.complaint_dao import save_complaint
+from app.dao.notifications_dao import dao_get_notification_history_by_reference
+from app.dao.service_callback_api_dao import (
+    get_service_complaint_callback_api_for_service,
+    get_service_delivery_status_callback_api_for_service,
 )
+from app.models import NOTIFICATION_PENDING, NOTIFICATION_SENDING, Complaint
 
 
 @notify_celery.task(bind=True, name="process-ses-result", max_retries=5, default_retry_delay=300)
 def process_ses_results(self, response):
     try:
-        ses_message = json.loads(response['Message'])
-        notification_type = ses_message['notificationType']
+        ses_message = json.loads(response["Message"])
+        notification_type = ses_message["notificationType"]
+        # TODO remove after smoke testing on prod is implemented
+        current_app.logger.info(
+            f"Attempting to process SES delivery status message "
+            f"from SNS with type: {notification_type} and body: {ses_message}"
+        )
         bounce_message = None
 
         if notification_type == 'Bounce':
-            notification_type, bounce_message = determine_notification_bounce_type(notification_type, ses_message)
+            bounce_message = determine_notification_bounce_type(ses_message)
         elif notification_type == 'Complaint':
             _check_and_queue_complaint_callback_task(*handle_complaint(ses_message))
             return True
 
-        aws_response_dict = get_aws_responses(notification_type)
+        aws_response_dict = get_aws_responses(ses_message)
 
-        notification_status = aws_response_dict['notification_status']
-        reference = ses_message['mail']['messageId']
+        notification_status = aws_response_dict["notification_status"]
+        reference = ses_message["mail"]["messageId"]
 
         try:
-            notification = notifications_dao.dao_get_notification_or_history_by_reference(reference=reference)
+            notification = notifications_dao.dao_get_notification_by_reference(reference)
         except NoResultFound:
-            message_time = iso8601.parse_date(ses_message['mail']['timestamp']).replace(tzinfo=None)
+            message_time = iso8601.parse_date(ses_message["mail"]["timestamp"]).replace(tzinfo=None)
             if datetime.utcnow() - message_time < timedelta(minutes=5):
                 current_app.logger.info(
-                    f"notification not found for reference: {reference} (update to {notification_status}). "
-                    f"Callback may have arrived before notification was persisted to the DB. Adding task to retry queue"
+                    f"Notification not found for reference: {reference}"
+                    f"(while attempting update to {notification_status}). "
+                    f"Callback may have arrived before notification was"
+                    f"persisted to the DB. Adding task to retry queue"
                 )
                 self.retry(queue=QueueNames.RETRY)
             else:
                 current_app.logger.warning(
-                    f"notification not found for reference: {reference} (update to {notification_status})"
+                    f"Notification not found for reference: {reference} "
+                    f"(while attempting update to {notification_status})"
                 )
             return
 
         if bounce_message:
             current_app.logger.info(f"SES bounce for notification ID {notification.id}: {bounce_message}")
 
-        if notification.status not in [NOTIFICATION_SENDING, NOTIFICATION_PENDING]:
+        if notification.status not in {NOTIFICATION_SENDING, NOTIFICATION_PENDING}:
             notifications_dao._duplicate_update_warning(
-                notification=notification,
-                status=notification_status
+                notification,
+                notification_status
             )
             return
-        else:
-            notifications_dao.dao_update_notifications_by_reference(
-                references=[reference],
-                update_dict={'status': notification_status}
+
+        notifications_dao._update_notification_status(
+            notification=notification,
+            status=notification_status,
+            provider_response=aws_response_dict["provider_response"],
+        )
+
+        if not aws_response_dict["success"]:
+            current_app.logger.info(
+                "SES delivery failed: notification id {} and reference {} has error found. Status {}".format(
+                    notification.id, reference, aws_response_dict["message"]
+                )
             )
-
-        statsd_client.incr('callback.ses.{}'.format(notification_status))
-
-        if notification.sent_at:
-            statsd_client.timing_with_dates(
-                f'callback.ses.{notification_status}.elapsed-time',
-                datetime.utcnow(),
-                notification.sent_at
+        else:
+            current_app.logger.info(
+                "SES callback return status of {} for notification: {}".format(notification_status, notification.id)
             )
 
         check_and_queue_callback_task(notification)
@@ -84,5 +100,139 @@ def process_ses_results(self, response):
         raise
 
     except Exception as e:
-        current_app.logger.exception('Error processing SES results: {}'.format(type(e)))
+        current_app.logger.exception("Error processing SES results: {}".format(type(e)))
         self.retry(queue=QueueNames.RETRY)
+
+
+def determine_notification_bounce_type(ses_message):
+    notification_type = ses_message["notificationType"]
+    if notification_type in ["Delivery", "Complaint"]:
+        return notification_type
+
+    if notification_type != "Bounce":
+        raise KeyError(f"Unhandled sns notification type {notification_type}")
+
+    remove_emails_from_bounce(ses_message)
+    current_app.logger.info("SES bounce dict: {}".format(json.dumps(ses_message).replace("{", "(").replace("}", ")")))
+    if ses_message["bounce"]["bounceType"] == "Permanent":
+        return "Permanent"
+    return "Temporary"
+
+
+def determine_notification_type(ses_message):
+    notification_type = ses_message["notificationType"]
+    if notification_type not in ["Bounce", "Complaint",   "Delivery"]:
+        raise KeyError(f"Unhandled sns notification type {notification_type}")
+    if notification_type == 'Bounce':
+        return determine_notification_bounce_type(ses_message)
+    return notification_type
+
+
+def _determine_provider_response(ses_message):
+    if ses_message["notificationType"] != "Bounce":
+        return None
+
+    bounce_type = ses_message["bounce"]["bounceType"]
+    bounce_subtype = ses_message["bounce"]["bounceSubType"]
+
+    # See https://docs.aws.amazon.com/ses/latest/DeveloperGuide/event-publishing-retrieving-sns-contents.html
+    if bounce_type == "Permanent" and bounce_subtype == "Suppressed":
+        return "The email address is on our email provider suppression list"
+    elif bounce_type == "Permanent" and bounce_subtype == "OnAccountSuppressionList":
+        return "The email address is on the GC Notify suppression list"
+    elif bounce_type == "Transient" and bounce_subtype == "AttachmentRejected":
+        return "The email was rejected because of its attachments"
+
+    return None
+
+
+def get_aws_responses(ses_message):
+    status = determine_notification_type(ses_message)
+
+    base = {
+        "Permanent": {
+            "message": "Hard bounced",
+            "success": False,
+            "notification_status": "permanent-failure",
+        },
+        "Temporary": {
+            "message": "Soft bounced",
+            "success": False,
+            "notification_status": "temporary-failure",
+        },
+        "Delivery": {
+            "message": "Delivered",
+            "success": True,
+            "notification_status": "delivered",
+        },
+        "Complaint": {
+            "message": "Complaint",
+            "success": True,
+            "notification_status": "delivered",
+        },
+    }[status]
+
+    base["provider_response"] = _determine_provider_response(ses_message)
+
+    return base
+
+
+def handle_complaint(ses_message):
+    recipient_email = remove_emails_from_complaint(ses_message)[0]
+    current_app.logger.info(
+        "Complaint from SES: \n{}".format(json.dumps(ses_message).replace("{", "(").replace("}", ")"))
+    )
+    try:
+        reference = ses_message["mail"]["messageId"]
+    except KeyError as e:
+        current_app.logger.exception(f"Complaint from SES failed to get reference from message with error: {e}")
+        return
+    notification = dao_get_notification_history_by_reference(reference)
+    ses_complaint = ses_message.get("complaint", None)
+
+    complaint = Complaint(
+        notification_id=notification.id,
+        service_id=notification.service_id,
+        ses_feedback_id=ses_complaint.get("feedbackId", None) if ses_complaint else None,
+        complaint_type=ses_complaint.get("complaintFeedbackType", None) if ses_complaint else None,
+        complaint_date=ses_complaint.get("timestamp", None) if ses_complaint else None,
+    )
+    save_complaint(complaint)
+    return complaint, notification, recipient_email
+
+
+def remove_mail_headers(dict_to_edit):
+    if dict_to_edit["mail"].get("headers"):
+        dict_to_edit["mail"].pop("headers")
+    if dict_to_edit["mail"].get("commonHeaders"):
+        dict_to_edit["mail"].pop("commonHeaders")
+
+
+def remove_emails_from_bounce(bounce_dict):
+    remove_mail_headers(bounce_dict)
+    bounce_dict["mail"].pop("destination", None)
+    bounce_dict["bounce"].pop("bouncedRecipients", None)
+
+
+def remove_emails_from_complaint(complaint_dict):
+    remove_mail_headers(complaint_dict)
+    complaint_dict["complaint"].pop("complainedRecipients")
+    return complaint_dict["mail"].pop("destination")
+
+
+def check_and_queue_callback_task(notification):
+    # queue callback task only if the service_callback_api exists
+    service_callback_api = get_service_delivery_status_callback_api_for_service(service_id=notification.service_id)
+    if service_callback_api:
+        notification_data = create_delivery_status_callback_data(notification, service_callback_api)
+        send_delivery_status_to_service.apply_async(
+            [str(notification.id), notification_data], queue=QueueNames.CALLBACKS
+        )
+
+
+def _check_and_queue_complaint_callback_task(complaint, notification, recipient):
+    # queue callback task only if the service_callback_api exists
+    service_callback_api = get_service_complaint_callback_api_for_service(service_id=notification.service_id)
+    if service_callback_api:
+        complaint_data = create_complaint_callback_data(complaint, notification, service_callback_api, recipient)
+        send_complaint_to_service.apply_async([complaint_data], queue=QueueNames.CALLBACKS)

app/celery/process_sms_client_response_tasks.py

@@ -1,94 +0,0 @@
-import uuid
-from datetime import datetime
-
-from flask import current_app
-from notifications_utils.template import SMSMessageTemplate
-
-from app import notify_celery, statsd_client
-from app.clients import ClientException
-from app.clients.sms.firetext import get_firetext_responses
-from app.clients.sms.mmg import get_mmg_responses
-from app.clients.sms.reach import get_reach_responses
-from app.dao import notifications_dao
-from app.dao.templates_dao import dao_get_template_by_id
-from app.models import NOTIFICATION_PENDING
-from app.notifications.notifications_ses_callback import (
-    check_and_queue_callback_task,
-)
-
-sms_response_mapper = {
-    'MMG': get_mmg_responses,
-    'Firetext': get_firetext_responses,
-    'Reach': get_reach_responses
-}
-
-
-@notify_celery.task(bind=True, name="process-sms-client-response", max_retries=5, default_retry_delay=300)
-def process_sms_client_response(self, status, provider_reference, client_name, detailed_status_code=None):
-    # validate reference
-    try:
-        uuid.UUID(provider_reference, version=4)
-    except ValueError as e:
-        current_app.logger.exception(f'{client_name} callback with invalid reference {provider_reference}')
-        raise e
-
-    response_parser = sms_response_mapper[client_name]
-
-    # validate status
-    try:
-        notification_status, detailed_status = response_parser(status, detailed_status_code)
-        current_app.logger.info(
-            f'{client_name} callback returned status of {notification_status}'
-            f'({status}): {detailed_status}({detailed_status_code}) for reference: {provider_reference}'
-        )
-    except KeyError:
-        _process_for_status(
-            notification_status='technical-failure',
-            client_name=client_name,
-            provider_reference=provider_reference
-        )
-        raise ClientException(f'{client_name} callback failed: status {status} not found.')
-
-    _process_for_status(
-        notification_status=notification_status,
-        client_name=client_name,
-        provider_reference=provider_reference,
-        detailed_status_code=detailed_status_code
-    )
-
-
-def _process_for_status(notification_status, client_name, provider_reference, detailed_status_code=None):
-    # record stats
-    notification = notifications_dao.update_notification_status_by_id(
-        notification_id=provider_reference,
-        status=notification_status,
-        sent_by=client_name.lower(),
-        detailed_status_code=detailed_status_code
-    )
-    if not notification:
-        return
-
-    statsd_client.incr('callback.{}.{}'.format(client_name.lower(), notification_status))
-
-    if notification.sent_at:
-        statsd_client.timing_with_dates(
-            f'callback.{client_name.lower()}.{notification_status}.elapsed-time',
-            datetime.utcnow(),
-            notification.sent_at
-        )
-
-    if notification.billable_units == 0:
-        service = notification.service
-        template_model = dao_get_template_by_id(notification.template_id, notification.template_version)
-
-        template = SMSMessageTemplate(
-            template_model.__dict__,
-            values=notification.personalisation,
-            prefix=service.name,
-            show_prefix=service.prefix_sms,
-        )
-        notification.billable_units = template.fragment_count
-        notifications_dao.dao_update_notification(notification)
-
-    if notification_status != NOTIFICATION_PENDING:
-        check_and_queue_callback_task(notification)

app/celery/provider_tasks.py

@@ -1,26 +1,72 @@
+from datetime import datetime, timedelta
+from time import time
+
 from flask import current_app
 from sqlalchemy.orm.exc import NoResultFound
 
-from app import notify_celery
+from app import aws_cloudwatch_client, notify_celery
 from app.clients.email import EmailClientNonRetryableException
 from app.clients.email.aws_ses import AwsSesClientThrottlingSendRateException
 from app.clients.sms import SmsClientResponseException
 from app.config import QueueNames
 from app.dao import notifications_dao
-from app.dao.notifications_dao import update_notification_status_by_id
+from app.dao.notifications_dao import (
+    sanitize_successful_notification_by_id,
+    update_notification_status_by_id,
+)
 from app.delivery import send_to_providers
 from app.exceptions import NotificationTechnicalFailureException
-from app.models import NOTIFICATION_TECHNICAL_FAILURE
+from app.models import (
+    NOTIFICATION_DELIVERED,
+    NOTIFICATION_FAILED,
+    NOTIFICATION_TECHNICAL_FAILURE,
+)
+
+
+@notify_celery.task(bind=True, name="check_sms_delivery_receipt", max_retries=48, default_retry_delay=300)
+def check_sms_delivery_receipt(self, message_id, notification_id, sent_at):
+    """
+    This is called after deliver_sms to check the status of the message. This uses the same number of
+    retries and the same delay period as deliver_sms.  In addition, this fires five minutes after
+    deliver_sms initially. So the idea is that most messages will succeed and show up in the logs quickly.
+    Other message will resolve successfully after a retry or to.  A few will fail but it will take up to
+    4 hours to know for sure.  The call to check_sms will raise an exception if neither a success nor a
+    failure appears in the cloudwatch logs, so this should keep retrying until the log appears, or until
+    we run out of retries.
+    """
+    status, provider_response = aws_cloudwatch_client.check_sms(message_id, notification_id, sent_at)
+    if status == 'success':
+        status = NOTIFICATION_DELIVERED
+    elif status == 'failure':
+        status = NOTIFICATION_FAILED
+    # if status is not success or failure the client raised an exception and this method will retry
+
+    if status == NOTIFICATION_DELIVERED:
+        sanitize_successful_notification_by_id(notification_id)
+        current_app.logger.info(f"Sanitized notification {notification_id} that was successfully delivered")
+    else:
+        update_notification_status_by_id(notification_id, status, provider_response=provider_response)
+        current_app.logger.info(f"Updated notification {notification_id} with response '{provider_response}'")
 
 
 @notify_celery.task(bind=True, name="deliver_sms", max_retries=48, default_retry_delay=300)
 def deliver_sms(self, notification_id):
     try:
+        # Get the time we are doing the sending, to minimize the time period we need to check over for receipt
+        now = round(time() * 1000)
         current_app.logger.info("Start sending SMS for notification id: {}".format(notification_id))
         notification = notifications_dao.get_notification_by_id(notification_id)
         if not notification:
             raise NoResultFound()
-        send_to_providers.send_sms_to_provider(notification)
+        message_id = send_to_providers.send_sms_to_provider(notification)
+        # We have to put it in UTC.  For other timezones, the delay
+        # will be ignored and it will fire immediately (although this probably only affects developer testing)
+        my_eta = datetime.utcnow() + timedelta(seconds=300)
+        check_sms_delivery_receipt.apply_async(
+            [message_id, notification_id, now],
+            eta=my_eta,
+            queue=QueueNames.CHECK_SMS
+        )
     except Exception as e:
         if isinstance(e, SmsClientResponseException):
             current_app.logger.warning(

app/celery/reporting_tasks.py

@@ -1,7 +1,6 @@
 from datetime import datetime, timedelta
 
 from flask import current_app
-from notifications_utils.timezones import convert_utc_to_bst
 
 from app import notify_celery
 from app.config import QueueNames
@@ -12,7 +11,7 @@ from app.dao.fact_billing_dao import (
 )
 from app.dao.fact_notification_status_dao import update_fact_notification_status
 from app.dao.notifications_dao import get_service_ids_with_notifications_on_date
-from app.models import EMAIL_TYPE, LETTER_TYPE, SMS_TYPE
+from app.models import EMAIL_TYPE, SMS_TYPE
 
 
 @notify_celery.task(name="create-nightly-billing")
@@ -21,11 +20,11 @@ def create_nightly_billing(day_start=None):
     # day_start is a datetime.date() object. e.g.
     # up to 4 days of data counting back from day_start is consolidated
     if day_start is None:
-        day_start = convert_utc_to_bst(datetime.utcnow()).date() - timedelta(days=1)
+        day_start = datetime.utcnow().date() - timedelta(days=1)
     else:
         # When calling the task its a string in the format of "YYYY-MM-DD"
         day_start = datetime.strptime(day_start, "%Y-%m-%d").date()
-    for i in range(0, 4):
+    for i in range(0, 10):
         process_day = (day_start - timedelta(days=i)).isoformat()
 
         create_nightly_billing_for_day.apply_async(
@@ -72,10 +71,6 @@ def create_nightly_notification_status():
         because all outstanding email / SMS are "timed out" after 3 days, and
         we reject delivery receipts after this point.
 
-      - Letter statuses don't change after 9 days. There's no "timeout" for
-        letters but this is the longest we've had to cope with in the past - due
-        to major issues with our print provider.
-
     Because the time range of the task exceeds the minimum possible retention
     period (3 days), we need to choose which table to query for each service.
 
@@ -87,10 +82,10 @@ def create_nightly_notification_status():
         mean the aggregated results are temporarily incorrect.
     """
 
-    yesterday = convert_utc_to_bst(datetime.utcnow()).date() - timedelta(days=1)
+    yesterday = datetime.utcnow().date() - timedelta(days=1)
 
-    for notification_type in [SMS_TYPE, EMAIL_TYPE, LETTER_TYPE]:
-        days = 10 if notification_type == LETTER_TYPE else 4
+    for notification_type in [SMS_TYPE, EMAIL_TYPE]:
+        days = 4
 
         for i in range(days):
             process_day = yesterday - timedelta(days=i)

app/celery/research_mode_tasks.py

@@ -1,43 +1,25 @@
 import json
-import random
-from datetime import datetime, timedelta
 
 from flask import current_app
-from notifications_utils.s3 import s3upload
 from requests import HTTPError, request
 
-from app import notify_celery
-from app.aws.s3 import file_exists
 from app.celery.process_ses_receipts_tasks import process_ses_results
 from app.config import QueueNames
+from app.dao.notifications_dao import get_notification_by_id
 from app.models import SMS_TYPE
 
-temp_fail = "7700900003"
-perm_fail = "7700900002"
-delivered = "7700900001"
+temp_fail = "2028675303"
+perm_fail = "2028675302"
+delivered = "2028675309"
 
 delivered_email = "delivered@simulator.notify"
 perm_fail_email = "perm-fail@simulator.notify"
 temp_fail_email = "temp-fail@simulator.notify"
 
 
-def send_sms_response(provider, reference, to):
-    if provider == "mmg":
-        body = mmg_callback(reference, to)
-        headers = {"Content-type": "application/json"}
-    else:
-        headers = {"Content-type": "application/x-www-form-urlencoded"}
-        body = firetext_callback(reference, to)
-        # to simulate getting a temporary_failure from firetext
-        # we need to send a pending status updated then a permanent-failure
-        if body['status'] == '2':  # pending status
-            make_request(SMS_TYPE, provider, body, headers)
-            # 1 is a declined status for firetext, will result in a temp-failure
-            body = {'mobile': to,
-                    'status': "1",
-                    'time': '2016-03-10 14:17:00',
-                    'reference': reference
-                    }
+def send_sms_response(provider, reference):
+    body = sns_callback(reference)
+    headers = {"Content-type": "application/json"}
 
     make_request(SMS_TYPE, provider, body, headers)
 
@@ -78,83 +60,15 @@ def make_request(notification_type, provider, data, headers):
     return response.json()
 
 
-def mmg_callback(notification_id, to):
-    """
-        status: 3 - delivered
-        status: 4 - expired (temp failure)
-        status: 5 - rejected (perm failure)
-    """
+def sns_callback(notification_id):
+    notification = get_notification_by_id(notification_id)
 
-    if to.strip().endswith(temp_fail):
-        status = "4"
-    elif to.strip().endswith(perm_fail):
-        status = "5"
-    else:
-        status = "3"
-
-    return json.dumps({"reference": "mmg_reference",
-                       "CID": str(notification_id),
-                       "MSISDN": to,
-                       "status": status,
-                       "deliverytime": "2016-04-05 16:01:07"})
-
-
-def firetext_callback(notification_id, to):
-    """
-        status: 0 - delivered
-        status: 1 - perm failure
-    """
-    if to.strip().endswith(perm_fail):
-        status = "1"
-    elif to.strip().endswith(temp_fail):
-        status = "2"
-    else:
-        status = "0"
-    return {
-        'mobile': to,
-        'status': status,
-        'time': '2016-03-10 14:17:00',
-        'reference': notification_id
-    }
-
-
-@notify_celery.task(bind=True, name="create-fake-letter-response-file", max_retries=5, default_retry_delay=300)
-def create_fake_letter_response_file(self, reference):
-    now = datetime.utcnow()
-    dvla_response_data = '{}|Sent|0|Sorted'.format(reference)
-
-    # try and find a filename that hasn't been taken yet - from a random time within the last 30 seconds
-    for i in sorted(range(30), key=lambda _: random.random()):
-        upload_file_name = 'NOTIFY-{}-RSP.TXT'.format((now - timedelta(seconds=i)).strftime('%Y%m%d%H%M%S'))
-        if not file_exists(current_app.config['DVLA_RESPONSE_BUCKET_NAME'], upload_file_name):
-            break
-    else:
-        raise ValueError(
-            'cant create fake letter response file for {} - too many files for that time already exist on s3'.format(
-                reference
-            )
-        )
-
-    s3upload(
-        filedata=dvla_response_data,
-        region=current_app.config['AWS_REGION'],
-        bucket_name=current_app.config['DVLA_RESPONSE_BUCKET_NAME'],
-        file_location=upload_file_name
-    )
-    current_app.logger.info("Fake DVLA response file {}, content [{}], uploaded to {}, created at {}".format(
-        upload_file_name, dvla_response_data, current_app.config['DVLA_RESPONSE_BUCKET_NAME'], now))
-
-    # on development we can't trigger SNS callbacks so we need to manually hit the DVLA callback endpoint
-    if current_app.config['NOTIFY_ENVIRONMENT'] == 'development':
-        make_request('letter', 'dvla', _fake_sns_s3_callback(upload_file_name), None)
-
-
-def _fake_sns_s3_callback(filename):
-    message_contents = '{"Records":[{"s3":{"object":{"key":"%s"}}}]}' % (filename)  # noqa
+    # This will only work if all notifications, including successful ones, are in the notifications table
+    # If we decide to delete successful notifications, we will have to get this from notifications history
     return json.dumps({
-        "Type": "Notification",
-        "MessageId": "some-message-id",
-        "Message": message_contents
+        "CID": str(notification_id),
+        "status": notification.status,
+        # "deliverytime": notification.completed_at
     })
 
 
@@ -164,7 +78,7 @@ def ses_notification_callback(reference):
             'processingTimeMillis': 2003,
             'recipients': ['success@simulator.amazonses.com'],
             'remoteMtaIp': '123.123.123.123',
-            'reportingMTA': 'a7-32.smtp-out.eu-west-1.amazonses.com',
+            'reportingMTA': 'a7-32.smtp-out.us-west-2.amazonses.com',
             'smtpResponse': '250 2.6.0 Message received',
             'timestamp': '2017-11-17T12:14:03.646Z'
         },
@@ -201,7 +115,7 @@ def ses_notification_callback(reference):
             'messageId': reference,
             'sendingAccountId': '12341234',
             'source': '"TEST" <TEST@notify.works>',
-            'sourceArn': 'arn:aws:ses:eu-west-1:12341234:identity/notify.works',
+            'sourceArn': 'arn:aws:ses:us-west-2:12341234:identity/notify.works',
             'sourceIp': '0.0.0.1',
             'timestamp': '2017-11-17T12:14:01.643Z'
         },
@@ -211,14 +125,14 @@ def ses_notification_callback(reference):
     return {
         'Type': 'Notification',
         'MessageId': '8e83c020-1234-1234-1234-92a8ee9baa0a',
-        'TopicArn': 'arn:aws:sns:eu-west-1:12341234:ses_notifications',
+        'TopicArn': 'arn:aws:sns:us-west-2:12341234:ses_notifications',
         'Subject': None,
         'Message': json.dumps(ses_message_body),
         'Timestamp': '2017-11-17T12:14:03.710Z',
         'SignatureVersion': '1',
         'Signature': '[REDACTED]',
-        'SigningCertUrl': 'https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-[REDACTED].pem',
-        'UnsubscribeUrl': 'https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=[REACTED]',
+        'SigningCertUrl': 'https://sns.us-west-2.amazonaws.com/SimpleNotificationService-[REDACTED].pem',
+        'UnsubscribeUrl': 'https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=[REACTED]',
         'MessageAttributes': {}
     }
 
@@ -244,7 +158,7 @@ def _ses_bounce_callback(reference, bounce_type):
             }],
             'feedbackId': '0102015fc9e676fb-12341234-1234-1234-1234-9301e86a4fa8-000000',
             'remoteMtaIp': '123.123.123.123',
-            'reportingMTA': 'dsn; a7-31.smtp-out.eu-west-1.amazonses.com',
+            'reportingMTA': 'dsn; a7-31.smtp-out.us-west-2.amazonses.com',
             'timestamp': '2017-11-17T12:14:05.131Z'
         },
         'mail': {
@@ -280,7 +194,7 @@ def _ses_bounce_callback(reference, bounce_type):
             'messageId': reference,
             'sendingAccountId': '12341234',
             'source': '"TEST" <TEST@notify.works>',
-            'sourceArn': 'arn:aws:ses:eu-west-1:12341234:identity/notify.works',
+            'sourceArn': 'arn:aws:ses:us-west-2:12341234:identity/notify.works',
             'sourceIp': '0.0.0.1',
             'timestamp': '2017-11-17T12:14:03.000Z'
         },
@@ -289,13 +203,13 @@ def _ses_bounce_callback(reference, bounce_type):
     return {
         'Type': 'Notification',
         'MessageId': '36e67c28-1234-1234-1234-2ea0172aa4a7',
-        'TopicArn': 'arn:aws:sns:eu-west-1:12341234:ses_notifications',
+        'TopicArn': 'arn:aws:sns:us-west-2:12341234:ses_notifications',
         'Subject': None,
         'Message': json.dumps(ses_message_body),
         'Timestamp': '2017-11-17T12:14:05.149Z',
         'SignatureVersion': '1',
         'Signature': '[REDACTED]',  # noqa
-        'SigningCertUrl': 'https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-[REDACTED]].pem',
-        'UnsubscribeUrl': 'https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=[REDACTED]]',
+        'SigningCertUrl': 'https://sns.us-west-2.amazonaws.com/SimpleNotificationService-[REDACTED]].pem',
+        'UnsubscribeUrl': 'https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=[REDACTED]]',
         'MessageAttributes': {}
     }

app/celery/scheduled_tasks.py

@@ -7,17 +7,14 @@ from notifications_utils.clients.zendesk.zendesk_client import (
 from sqlalchemy import between
 from sqlalchemy.exc import SQLAlchemyError
 
-from app import db, notify_celery, zendesk_client
-from app.aws import s3
-from app.celery.broadcast_message_tasks import trigger_link_test
-from app.celery.letters_pdf_tasks import get_pdf_for_templated_letter
+from app import notify_celery, zendesk_client
 from app.celery.tasks import (
     get_recipient_csv_and_template_and_sender_id,
     process_incomplete_jobs,
     process_job,
     process_row,
 )
-from app.config import QueueNames, TaskNames
+from app.config import QueueNames
 from app.dao.invited_org_user_dao import (
     delete_org_invitations_created_more_than_two_days_ago,
 )
@@ -30,31 +27,21 @@ from app.dao.jobs_dao import (
     find_jobs_with_missing_rows,
     find_missing_row_for_job,
 )
-from app.dao.notifications_dao import (
-    dao_old_letters_with_created_status,
-    dao_precompiled_letters_still_pending_virus_check,
-    is_delivery_slow_for_providers,
-    letters_missing_from_sending_bucket,
-    notifications_not_yet_sent,
-)
+from app.dao.notifications_dao import notifications_not_yet_sent
 from app.dao.provider_details_dao import (
     dao_adjust_provider_priority_back_to_resting_points,
-    dao_reduce_sms_provider_priority,
 )
 from app.dao.services_dao import (
     dao_find_services_sending_to_tv_numbers,
     dao_find_services_with_high_failure_rates,
 )
 from app.dao.users_dao import delete_codes_older_created_more_than_a_day_ago
-from app.letters.utils import generate_letter_pdf_filename
 from app.models import (
     EMAIL_TYPE,
     JOB_STATUS_ERROR,
     JOB_STATUS_IN_PROGRESS,
     JOB_STATUS_PENDING,
     SMS_TYPE,
-    BroadcastMessage,
-    BroadcastStatusType,
     Job,
 )
 from app.notifications.process_notifications import send_notification_to_queue
@@ -98,28 +85,6 @@ def delete_invitations():
         raise
 
 
-@notify_celery.task(name='switch-current-sms-provider-on-slow-delivery')
-def switch_current_sms_provider_on_slow_delivery():
-    """
-    Reduce provider's priority if at least 30% of notifications took more than four minutes to be delivered
-    in the last ten minutes. If both providers are slow, don't do anything. If we changed the providers in the
-    last ten minutes, then don't update them again either.
-    """
-    slow_delivery_notifications = is_delivery_slow_for_providers(
-        threshold=0.3,
-        created_at=datetime.utcnow() - timedelta(minutes=10),
-        delivery_time=timedelta(minutes=4),
-    )
-
-    # only adjust if some values are true and some are false - ie, don't adjust if all providers are fast or
-    # all providers are slow
-    if len(set(slow_delivery_notifications.values())) != 1:
-        for provider_name, is_slow in slow_delivery_notifications.items():
-            if is_slow:
-                current_app.logger.warning('Slow delivery notifications detected for provider {}'.format(provider_name))
-                dao_reduce_sms_provider_priority(provider_name, time_threshold=timedelta(minutes=10))
-
-
 @notify_celery.task(name='tend-providers-back-to-middle')
 def tend_providers_back_to_middle():
     dao_adjust_provider_priority_back_to_resting_points()
@@ -192,88 +157,6 @@ def replay_created_notifications():
         for n in notifications_to_resend:
             send_notification_to_queue(notification=n, research_mode=n.service.research_mode)
 
-    # if the letter has not be send after an hour, then create a zendesk ticket
-    letters = letters_missing_from_sending_bucket(resend_created_notifications_older_than)
-
-    if len(letters) > 0:
-        msg = "{} letters were created over an hour ago, " \
-              "but do not have an updated_at timestamp or billable units. " \
-              "\n Creating app.celery.letters_pdf_tasks.create_letters tasks to upload letter to S3 " \
-              "and update notifications for the following notification ids: " \
-              "\n {}".format(len(letters), [x.id for x in letters])
-
-        current_app.logger.info(msg)
-        for letter in letters:
-            get_pdf_for_templated_letter.apply_async([str(letter.id)], queue=QueueNames.CREATE_LETTERS_PDF)
-
-
-@notify_celery.task(name='check-if-letters-still-pending-virus-check')
-def check_if_letters_still_pending_virus_check():
-    letters = []
-
-    for letter in dao_precompiled_letters_still_pending_virus_check():
-        # find letter in the scan bucket
-        filename = generate_letter_pdf_filename(
-            letter.reference,
-            letter.created_at,
-            ignore_folder=True,
-            postage=letter.postage
-        )
-
-        if s3.file_exists(current_app.config['LETTERS_SCAN_BUCKET_NAME'], filename):
-            current_app.logger.warning(
-                f'Letter id {letter.id} got stuck in pending-virus-check. Sending off for scan again.'
-            )
-            notify_celery.send_task(
-                name=TaskNames.SCAN_FILE,
-                kwargs={'filename': filename},
-                queue=QueueNames.ANTIVIRUS,
-            )
-        else:
-            letters.append(letter)
-
-    if len(letters) > 0:
-        letter_ids = [(str(letter.id), letter.reference) for letter in letters]
-
-        msg = f"""{len(letters)} precompiled letters have been pending-virus-check for over 90 minutes.
-            We couldn't find them in the scan bucket. We'll need to find out where the files are and kick them off
-            again or move them to technical failure.
-
-            Notifications: {sorted(letter_ids)}"""
-
-        if current_app.config['NOTIFY_ENVIRONMENT'] in ['live', 'production', 'test']:
-            ticket = NotifySupportTicket(
-                subject=f"[{current_app.config['NOTIFY_ENVIRONMENT']}] Letters still pending virus check",
-                message=msg,
-                ticket_type=NotifySupportTicket.TYPE_INCIDENT,
-                technical_ticket=True,
-                ticket_categories=['notify_letters']
-            )
-            zendesk_client.send_ticket_to_zendesk(ticket)
-            current_app.logger.error(msg)
-
-
-@notify_celery.task(name='check-if-letters-still-in-created')
-def check_if_letters_still_in_created():
-    letters = dao_old_letters_with_created_status()
-
-    if len(letters) > 0:
-        msg = "{} letters were created before 17.30 yesterday and still have 'created' status. " \
-              "Follow runbook to resolve: " \
-              "https://github.com/alphagov/notifications-manuals/wiki/Support-Runbook" \
-              "#deal-with-Letters-still-in-created.".format(len(letters))
-
-        if current_app.config['NOTIFY_ENVIRONMENT'] in ['live', 'production', 'test']:
-            ticket = NotifySupportTicket(
-                subject=f"[{current_app.config['NOTIFY_ENVIRONMENT']}] Letters still in 'created' status",
-                message=msg,
-                ticket_type=NotifySupportTicket.TYPE_INCIDENT,
-                technical_ticket=True,
-                ticket_categories=['notify_letters']
-            )
-            zendesk_client.send_ticket_to_zendesk(ticket)
-            current_app.logger.error(msg)
-
 
 @notify_celery.task(name='check-for-missing-rows-in-completed-jobs')
 def check_for_missing_rows_in_completed_jobs():
@@ -333,37 +216,3 @@ def check_for_services_with_high_failure_rates_or_sending_to_tv_numbers():
                 technical_ticket=True
             )
             zendesk_client.send_ticket_to_zendesk(ticket)
-
-
-@notify_celery.task(name='trigger-link-tests')
-def trigger_link_tests():
-    if current_app.config['CBC_PROXY_ENABLED']:
-        for cbc_name in current_app.config['ENABLED_CBCS']:
-            trigger_link_test.apply_async(kwargs={'provider': cbc_name}, queue=QueueNames.BROADCASTS)
-
-
-@notify_celery.task(name='auto-expire-broadcast-messages')
-def auto_expire_broadcast_messages():
-    expired_broadcasts = BroadcastMessage.query.filter(
-        BroadcastMessage.finishes_at <= datetime.now(),
-        BroadcastMessage.status == BroadcastStatusType.BROADCASTING,
-    ).all()
-
-    for broadcast in expired_broadcasts:
-        broadcast.status = BroadcastStatusType.COMPLETED
-
-    db.session.commit()
-
-    if expired_broadcasts:
-        notify_celery.send_task(
-            name=TaskNames.PUBLISH_GOVUK_ALERTS,
-            queue=QueueNames.GOVUK_ALERTS
-        )
-
-
-@notify_celery.task(name='remove-yesterdays-planned-tests-on-govuk-alerts')
-def remove_yesterdays_planned_tests_on_govuk_alerts():
-    notify_celery.send_task(
-        name=TaskNames.PUBLISH_GOVUK_ALERTS,
-        queue=QueueNames.GOVUK_ALERTS
-    )

app/celery/service_callback_tasks.py

@@ -114,6 +114,7 @@ def create_delivery_status_callback_data(notification, service_callback_api):
         "notification_client_reference": notification.client_reference,
         "notification_to": notification.to,
         "notification_status": notification.status,
+        "notification_provider_response": notification.provider_response,  # TODO do we test for provider_response?
         "notification_created_at": notification.created_at.strftime(DATETIME_FORMAT),
         "notification_updated_at":
             notification.updated_at.strftime(DATETIME_FORMAT) if notification.updated_at else None,

app/celery/tasks.py

@@ -1,63 +1,39 @@
 import json
-from collections import defaultdict, namedtuple
 from datetime import datetime
 
 from flask import current_app
-from notifications_utils.insensitive_dict import InsensitiveDict
-from notifications_utils.postal_address import PostalAddress
 from notifications_utils.recipients import RecipientCSV
-from notifications_utils.timezones import convert_utc_to_bst
 from requests import HTTPError, RequestException, request
 from sqlalchemy.exc import IntegrityError, SQLAlchemyError
 
-from app import create_random_identifier, create_uuid, encryption, notify_celery
+from app import create_uuid, encryption, notify_celery
 from app.aws import s3
-from app.celery import letters_pdf_tasks, provider_tasks, research_mode_tasks
+from app.celery import provider_tasks
 from app.config import QueueNames
-from app.dao.daily_sorted_letter_dao import (
-    dao_create_or_update_daily_sorted_letter,
-)
 from app.dao.inbound_sms_dao import dao_get_inbound_sms_by_id
 from app.dao.jobs_dao import dao_get_job_by_id, dao_update_job
 from app.dao.notifications_dao import (
     dao_get_last_notification_added_for_job_id,
-    dao_get_notification_or_history_by_reference,
-    dao_update_notifications_by_reference,
     get_notification_by_id,
-    update_notification_status_by_reference,
 )
-from app.dao.provider_details_dao import (
-    get_provider_details_by_notification_type,
-)
-from app.dao.returned_letters_dao import insert_or_update_returned_letters
 from app.dao.service_email_reply_to_dao import dao_get_reply_to_by_id
 from app.dao.service_inbound_api_dao import get_service_inbound_api_for_service
 from app.dao.service_sms_sender_dao import dao_get_service_sms_senders_by_id
 from app.dao.templates_dao import dao_get_template_by_id
-from app.exceptions import DVLAException, NotificationTechnicalFailureException
 from app.models import (
-    DVLA_RESPONSE_STATUS_SENT,
     EMAIL_TYPE,
     JOB_STATUS_CANCELLED,
     JOB_STATUS_FINISHED,
     JOB_STATUS_IN_PROGRESS,
     JOB_STATUS_PENDING,
     KEY_TYPE_NORMAL,
-    LETTER_TYPE,
-    NOTIFICATION_CREATED,
-    NOTIFICATION_DELIVERED,
-    NOTIFICATION_RETURNED_LETTER,
-    NOTIFICATION_SENDING,
-    NOTIFICATION_TECHNICAL_FAILURE,
-    NOTIFICATION_TEMPORARY_FAILURE,
     SMS_TYPE,
-    DailySortedLetter,
 )
 from app.notifications.process_notifications import persist_notification
 from app.notifications.validators import check_service_over_daily_message_limit
 from app.serialised_models import SerialisedService, SerialisedTemplate
 from app.service.utils import service_allowed_to_send_to
-from app.utils import DATETIME_FORMAT, get_reference_from_personalisation
+from app.utils import DATETIME_FORMAT
 from app.v2.errors import TooManyRequestsError
 
 
@@ -136,8 +112,7 @@ def process_row(row, template, job, service, sender_id=None):
 
     send_fns = {
         SMS_TYPE: save_sms,
-        EMAIL_TYPE: save_email,
-        LETTER_TYPE: save_letter
+        EMAIL_TYPE: save_email
     }
 
     send_fn = send_fns[template_type]
@@ -341,103 +316,6 @@ def save_api_email_or_sms(self, encrypted_notification):
             current_app.logger.error(f"Max retry failed Failed to persist notification {notification['id']}")
 
 
-@notify_celery.task(bind=True, name="save-letter", max_retries=5, default_retry_delay=300)
-def save_letter(
-        self,
-        service_id,
-        notification_id,
-        encrypted_notification,
-):
-    notification = encryption.decrypt(encrypted_notification)
-
-    postal_address = PostalAddress.from_personalisation(
-        InsensitiveDict(notification['personalisation'])
-    )
-
-    service = SerialisedService.from_id(service_id)
-    template = SerialisedTemplate.from_id_and_service_id(
-        notification['template'],
-        service_id=service.id,
-        version=notification['template_version'],
-    )
-
-    try:
-        # if we don't want to actually send the letter, then start it off in SENDING so we don't pick it up
-        status = NOTIFICATION_CREATED if not service.research_mode else NOTIFICATION_SENDING
-
-        saved_notification = persist_notification(
-            template_id=notification['template'],
-            template_version=notification['template_version'],
-            postage=postal_address.postage if postal_address.international else template.postage,
-            recipient=postal_address.normalised,
-            service=service,
-            personalisation=notification['personalisation'],
-            notification_type=LETTER_TYPE,
-            api_key_id=None,
-            key_type=KEY_TYPE_NORMAL,
-            created_at=datetime.utcnow(),
-            job_id=notification['job'],
-            job_row_number=notification['row_number'],
-            notification_id=notification_id,
-            reference=create_random_identifier(),
-            client_reference=get_reference_from_personalisation(notification['personalisation']),
-            reply_to_text=template.reply_to_text,
-            status=status
-        )
-
-        if not service.research_mode:
-            letters_pdf_tasks.get_pdf_for_templated_letter.apply_async(
-                [str(saved_notification.id)],
-                queue=QueueNames.CREATE_LETTERS_PDF
-            )
-        elif current_app.config['NOTIFY_ENVIRONMENT'] in ['preview', 'development']:
-            research_mode_tasks.create_fake_letter_response_file.apply_async(
-                (saved_notification.reference,),
-                queue=QueueNames.RESEARCH_MODE
-            )
-        else:
-            update_notification_status_by_reference(saved_notification.reference, 'delivered')
-
-        current_app.logger.debug("Letter {} created at {}".format(saved_notification.id, saved_notification.created_at))
-    except SQLAlchemyError as e:
-        handle_exception(self, notification, notification_id, e)
-
-
-@notify_celery.task(bind=True, name='update-letter-notifications-to-sent')
-def update_letter_notifications_to_sent_to_dvla(self, notification_references):
-    # This task will be called by the FTP app to update notifications as sent to DVLA
-    provider = get_provider_details_by_notification_type(LETTER_TYPE)[0]
-
-    updated_count, _ = dao_update_notifications_by_reference(
-        notification_references,
-        {
-            'status': NOTIFICATION_SENDING,
-            'sent_by': provider.identifier,
-            'sent_at': datetime.utcnow(),
-            'updated_at': datetime.utcnow()
-        }
-    )
-
-    current_app.logger.info("Updated {} letter notifications to sending".format(updated_count))
-
-
-@notify_celery.task(bind=True, name='update-letter-notifications-to-error')
-def update_letter_notifications_to_error(self, notification_references):
-    # This task will be called by the FTP app to update notifications as sent to DVLA
-
-    updated_count, _ = dao_update_notifications_by_reference(
-        notification_references,
-        {
-            'status': NOTIFICATION_TECHNICAL_FAILURE,
-            'updated_at': datetime.utcnow()
-        }
-    )
-    message = "Updated {} letter notifications to technical-failure with references {}".format(
-        updated_count, notification_references
-    )
-    raise NotificationTechnicalFailureException(message)
-
-
 def handle_exception(task, notification, notification_id, exc):
     if not get_notification_by_id(notification_id):
         retry_msg = '{task} notification for job {job} row number {row} and notification id {noti}'.format(
@@ -449,6 +327,7 @@ def handle_exception(task, notification, notification_id, exc):
         # Sometimes, SQS plays the same message twice. We should be able to catch an IntegrityError, but it seems
         # SQLAlchemy is throwing a FlushError. So we check if the notification id already exists then do not
         # send to the retry queue.
+        # This probably (hopefully) is not an issue with Redis as the celery backing store
         current_app.logger.exception('Retry' + retry_msg)
         try:
             task.retry(queue=QueueNames.RETRY, exc=exc)
@@ -456,108 +335,6 @@ def handle_exception(task, notification, notification_id, exc):
             current_app.logger.error('Max retry failed' + retry_msg)
 
 
-@notify_celery.task(bind=True, name='update-letter-notifications-statuses')
-def update_letter_notifications_statuses(self, filename):
-    notification_updates = parse_dvla_file(filename)
-
-    temporary_failures = []
-
-    for update in notification_updates:
-        check_billable_units(update)
-        update_letter_notification(filename, temporary_failures, update)
-    if temporary_failures:
-        # This will alert Notify that DVLA was unable to deliver the letters, we need to investigate
-        message = "DVLA response file: {filename} has failed letters with notification.reference {failures}" \
-            .format(filename=filename, failures=temporary_failures)
-        raise DVLAException(message)
-
-
-@notify_celery.task(bind=True, name="record-daily-sorted-counts")
-def record_daily_sorted_counts(self, filename):
-    sorted_letter_counts = defaultdict(int)
-    notification_updates = parse_dvla_file(filename)
-    for update in notification_updates:
-        sorted_letter_counts[update.cost_threshold.lower()] += 1
-
-    unknown_status = sorted_letter_counts.keys() - {'unsorted', 'sorted'}
-    if unknown_status:
-        message = 'DVLA response file: {} contains unknown Sorted status {}'.format(
-            filename, unknown_status.__repr__()
-        )
-        raise DVLAException(message)
-
-    billing_date = get_billing_date_in_bst_from_filename(filename)
-    persist_daily_sorted_letter_counts(day=billing_date,
-                                       file_name=filename,
-                                       sorted_letter_counts=sorted_letter_counts)
-
-
-def parse_dvla_file(filename):
-    bucket_location = '{}-ftp'.format(current_app.config['NOTIFY_EMAIL_DOMAIN'])
-    response_file_content = s3.get_s3_file(bucket_location, filename)
-
-    try:
-        return process_updates_from_file(response_file_content)
-    except TypeError:
-        raise DVLAException('DVLA response file: {} has an invalid format'.format(filename))
-
-
-def get_billing_date_in_bst_from_filename(filename):
-    # exclude seconds from the date since we don't need it. We got a date ending in 60 second - which is not valid.
-    datetime_string = filename.split('-')[1][:-2]
-    datetime_obj = datetime.strptime(datetime_string, '%Y%m%d%H%M')
-    return convert_utc_to_bst(datetime_obj).date()
-
-
-def persist_daily_sorted_letter_counts(day, file_name, sorted_letter_counts):
-    daily_letter_count = DailySortedLetter(
-        billing_day=day,
-        file_name=file_name,
-        unsorted_count=sorted_letter_counts['unsorted'],
-        sorted_count=sorted_letter_counts['sorted']
-    )
-    dao_create_or_update_daily_sorted_letter(daily_letter_count)
-
-
-def process_updates_from_file(response_file):
-    NotificationUpdate = namedtuple('NotificationUpdate', ['reference', 'status', 'page_count', 'cost_threshold'])
-    notification_updates = [NotificationUpdate(*line.split('|')) for line in response_file.splitlines()]
-    return notification_updates
-
-
-def update_letter_notification(filename, temporary_failures, update):
-    if update.status == DVLA_RESPONSE_STATUS_SENT:
-        status = NOTIFICATION_DELIVERED
-    else:
-        status = NOTIFICATION_TEMPORARY_FAILURE
-        temporary_failures.append(update.reference)
-
-    updated_count, _ = dao_update_notifications_by_reference(
-        references=[update.reference],
-        update_dict={"status": status,
-                     "updated_at": datetime.utcnow()
-                     }
-    )
-
-    if not updated_count:
-        msg = "Update letter notification file {filename} failed: notification either not found " \
-              "or already updated from delivered. Status {status} for notification reference {reference}".format(
-                  filename=filename, status=status, reference=update.reference)
-        current_app.logger.info(msg)
-
-
-def check_billable_units(notification_update):
-    notification = dao_get_notification_or_history_by_reference(notification_update.reference)
-
-    if int(notification_update.page_count) != notification.billable_units:
-        msg = 'Notification with id {} has {} billable_units but DVLA says page count is {}'.format(
-            notification.id, notification.billable_units, notification_update.page_count)
-        try:
-            raise DVLAException(msg)
-        except DVLAException:
-            current_app.logger.exception(msg)
-
-
 @notify_celery.task(bind=True, name="send-inbound-sms", max_retries=5, default_retry_delay=300)
 def send_inbound_sms_to_service(self, inbound_sms_id, service_id):
     inbound_api = get_service_inbound_api_for_service(service_id=service_id)
@@ -646,19 +423,3 @@ def process_incomplete_job(job_id):
             process_row(row, template, job, job.service, sender_id=sender_id)
 
     job_complete(job, resumed=True)
-
-
-@notify_celery.task(name='process-returned-letters-list')
-def process_returned_letters_list(notification_references):
-    updated, updated_history = dao_update_notifications_by_reference(
-        notification_references,
-        {"status": NOTIFICATION_RETURNED_LETTER}
-    )
-
-    insert_or_update_returned_letters(notification_references)
-
-    current_app.logger.info(
-        "Updated {} letter notifications ({} history notifications, from {} references) to returned-letter".format(
-            updated, updated_history, len(notification_references)
-        )
-    )

app/clients/__init__.py

@@ -35,7 +35,7 @@ class NotificationProviderClients(object):
         return self.email_clients.get(name)
 
     def get_client_by_name_and_type(self, name, notification_type):
-        assert notification_type in ['email', 'sms']
+        assert notification_type in ['email', 'sms']  # nosec B101
 
         if notification_type == 'email':
             return self.get_email_client(name)

app/clients/cbc_proxy.py

@@ -1,299 +0,0 @@
-import json
-import uuid
-from abc import ABC, abstractmethod
-
-import boto3
-import botocore
-from flask import current_app
-from notifications_utils.template import non_gsm_characters
-from sqlalchemy.schema import Sequence
-
-from app.config import BroadcastProvider
-from app.utils import DATETIME_FORMAT, format_sequential_number
-
-# The variable names in this file have specific meaning in a CAP message
-#
-# identifier is a unique field for each CAP message
-#
-# headline is a field which we are not sure if we will use
-#
-# description is the body of the message
-
-# areas is a list of dicts, with the following items
-# * description is a string which populates the areaDesc field
-# * polygon is a list of lat/long pairs
-#
-# previous_provider_messages is a list of previous events (models.py::BroadcastProviderMessage)
-# ie a Cancel message would have a unique event but have the event of
-#    the preceeding Alert message in the previous_provider_messages field
-
-
-class CBCProxyRetryableException(Exception):
-    pass
-
-
-class CBCProxyClient:
-    _lambda_client = None
-
-    def init_app(self, app):
-        if app.config.get('CBC_PROXY_ENABLED'):
-            self._lambda_client = boto3.client(
-                'lambda',
-                region_name='eu-west-2',
-                aws_access_key_id=app.config['CBC_PROXY_AWS_ACCESS_KEY_ID'],
-                aws_secret_access_key=app.config['CBC_PROXY_AWS_SECRET_ACCESS_KEY'],
-            )
-
-    def get_proxy(self, provider):
-        proxy_classes = {
-            BroadcastProvider.EE: CBCProxyEE,
-            BroadcastProvider.THREE: CBCProxyThree,
-            BroadcastProvider.O2: CBCProxyO2,
-            BroadcastProvider.VODAFONE: CBCProxyVodafone,
-        }
-        return proxy_classes[provider](self._lambda_client)
-
-
-class CBCProxyClientBase(ABC):
-    @property
-    @abstractmethod
-    def lambda_name(self):
-        pass
-
-    @property
-    @abstractmethod
-    def failover_lambda_name(self):
-        pass
-
-    @property
-    @abstractmethod
-    def LANGUAGE_ENGLISH(self):
-        pass
-
-    @property
-    @abstractmethod
-    def LANGUAGE_WELSH(self):
-        pass
-
-    def __init__(self, lambda_client):
-        self._lambda_client = lambda_client
-
-    def send_link_test(self):
-        self._send_link_test(self.lambda_name)
-        self._send_link_test(self.failover_lambda_name)
-
-    def _send_link_test(
-        self,
-        lambda_name,
-    ): pass
-
-    def create_and_send_broadcast(
-        self, identifier, headline, description, areas, sent, expires, channel, message_number=None
-    ):
-        pass
-
-    # We have not implementated updating a broadcast
-    def update_and_send_broadcast(
-        self,
-        identifier, previous_provider_messages, headline, description, areas,
-        sent, expires, channel, message_number=None
-    ):
-        pass
-
-    def cancel_broadcast(
-        self,
-        identifier, previous_provider_messages, headline, description, areas,
-        sent, expires, message_number=None
-    ):
-        pass
-
-    def _invoke_lambda_with_failover(self, payload):
-        result = self._invoke_lambda(self.lambda_name, payload)
-
-        if not result:
-            failover_result = self._invoke_lambda(self.failover_lambda_name, payload)
-            if not failover_result:
-                raise CBCProxyRetryableException(
-                    f'Lambda failed for both {self.lambda_name} and {self.failover_lambda_name}'
-                )
-
-        return result
-
-    def _invoke_lambda(self, lambda_name, payload):
-        payload_bytes = bytes(json.dumps(payload), encoding='utf8')
-        try:
-            current_app.logger.info(
-                f"Calling lambda {lambda_name} with payload {str(payload)[:1000]}"
-            )
-
-            result = self._lambda_client.invoke(
-                FunctionName=lambda_name,
-                InvocationType='RequestResponse',
-                Payload=payload_bytes,
-            )
-        except botocore.exceptions.ClientError:
-            current_app.logger.exception(f'Boto ClientError calling lambda {lambda_name}')
-            success = False
-            return success
-
-        if result['StatusCode'] > 299:
-            current_app.logger.info(
-                f"Error calling lambda {lambda_name} with status code { result['StatusCode']}, {result.get('Payload')}"
-            )
-            success = False
-
-        elif 'FunctionError' in result:
-            current_app.logger.info(
-                f"Error calling lambda {lambda_name} with function error { result['Payload'].read() }"
-            )
-            success = False
-
-        else:
-            success = True
-
-        return success
-
-    def infer_language_from(self, content):
-        if non_gsm_characters(content):
-            return self.LANGUAGE_WELSH
-        return self.LANGUAGE_ENGLISH
-
-
-class CBCProxyOne2ManyClient(CBCProxyClientBase):
-    LANGUAGE_ENGLISH = 'en-GB'
-    LANGUAGE_WELSH = 'cy-GB'
-
-    def _send_link_test(
-        self,
-        lambda_name,
-    ):
-        """
-        link test - open up a connection to a specific provider, and send them an xml payload with a <msgType> of
-        test.
-        """
-        payload = {
-            'message_type': 'test',
-            'identifier': str(uuid.uuid4()),
-            'message_format': 'cap'
-        }
-
-        self._invoke_lambda(lambda_name=lambda_name, payload=payload)
-
-    def create_and_send_broadcast(
-        self, identifier, headline, description, areas, sent, expires, channel, message_number=None
-    ):
-        payload = {
-            'message_type': 'alert',
-            'identifier': identifier,
-            'message_format': 'cap',
-            'headline': headline,
-            'description': description,
-            'areas': areas,
-            'sent': sent,
-            'expires': expires,
-            'language': self.infer_language_from(description),
-            'channel': channel,
-        }
-        self._invoke_lambda_with_failover(payload=payload)
-
-    def cancel_broadcast(
-        self,
-        identifier, previous_provider_messages,
-        sent, message_number=None
-    ):
-        payload = {
-            'message_type': 'cancel',
-            'identifier': identifier,
-            'message_format': 'cap',
-            "references": [
-                {
-                    "message_id": str(message.id),
-                    "sent": message.created_at.strftime(DATETIME_FORMAT)
-                } for message in previous_provider_messages
-            ],
-            'sent': sent,
-        }
-        self._invoke_lambda_with_failover(payload=payload)
-
-
-class CBCProxyEE(CBCProxyOne2ManyClient):
-    lambda_name = 'ee-1-proxy'
-    failover_lambda_name = 'ee-2-proxy'
-
-
-class CBCProxyThree(CBCProxyOne2ManyClient):
-    lambda_name = 'three-1-proxy'
-    failover_lambda_name = 'three-2-proxy'
-
-
-class CBCProxyO2(CBCProxyOne2ManyClient):
-    lambda_name = 'o2-1-proxy'
-    failover_lambda_name = 'o2-2-proxy'
-
-
-class CBCProxyVodafone(CBCProxyClientBase):
-    lambda_name = 'vodafone-1-proxy'
-    failover_lambda_name = 'vodafone-2-proxy'
-
-    LANGUAGE_ENGLISH = 'English'
-    LANGUAGE_WELSH = 'Welsh'
-
-    def _send_link_test(
-        self,
-        lambda_name,
-    ):
-        """
-        link test - open up a connection to a specific provider, and send them an xml payload with a <msgType> of
-        test.
-        """
-        from app import db
-        sequence = Sequence('broadcast_provider_message_number_seq')
-        sequential_number = db.session.connection().execute(sequence)
-        formatted_seq_number = format_sequential_number(sequential_number)
-
-        payload = {
-            'message_type': 'test',
-            'identifier': str(uuid.uuid4()),
-            'message_number': formatted_seq_number,
-            'message_format': 'ibag'
-        }
-
-        self._invoke_lambda(lambda_name=lambda_name, payload=payload)
-
-    def create_and_send_broadcast(
-        self, identifier, message_number, headline, description, areas, sent, expires, channel
-    ):
-        payload = {
-            'message_type': 'alert',
-            'identifier': identifier,
-            'message_number': message_number,
-            'message_format': 'ibag',
-            'headline': headline,
-            'description': description,
-            'areas': areas,
-            'sent': sent,
-            'expires': expires,
-            'language': self.infer_language_from(description),
-            'channel': channel,
-        }
-        self._invoke_lambda_with_failover(payload=payload)
-
-    def cancel_broadcast(
-        self, identifier, previous_provider_messages, sent, message_number
-    ):
-
-        payload = {
-            'message_type': 'cancel',
-            'identifier': identifier,
-            'message_number': message_number,
-            'message_format': 'ibag',
-            "references": [
-                {
-                    "message_id": str(message.id),
-                    "message_number": format_sequential_number(message.message_number),
-                    "sent": message.created_at.strftime(DATETIME_FORMAT)
-                } for message in previous_provider_messages
-            ],
-            'sent': sent,
-        }
-        self._invoke_lambda_with_failover(payload=payload)

app/clients/cloudwatch/aws_cloudwatch.py

@@ -0,0 +1,89 @@
+import json
+import re
+import time
+
+from boto3 import client
+
+from app.clients import Client
+from app.cloudfoundry_config import cloud_config
+
+
+class AwsCloudwatchClient(Client):
+    """
+    This client is responsible for retrieving sms delivery receipts from cloudwatch.
+    """
+
+    def init_app(self, current_app, *args, **kwargs):
+        self._client = client(
+            "logs",
+            region_name=cloud_config.sns_region,
+            aws_access_key_id=cloud_config.sns_access_key,
+            aws_secret_access_key=cloud_config.sns_secret_key
+        )
+        super(Client, self).__init__(*args, **kwargs)
+        self.current_app = current_app
+        self._valid_sender_regex = re.compile(r"^\+?\d{5,14}$")
+
+    @property
+    def name(self):
+        return 'cloudwatch'
+
+    def _get_log(self, my_filter, log_group_name, sent_at):
+
+        # Check all cloudwatch logs from the time the notification was sent (currently 5 minutes previously) until now
+        now = round(time.time() * 1000)
+        beginning = sent_at
+        next_token = None
+        all_log_events = []
+        while True:
+            if next_token:
+                response = self._client.filter_log_events(
+                    logGroupName=log_group_name,
+                    filterPattern=my_filter,
+                    nextToken=next_token,
+                    startTime=beginning,
+                    endTime=now
+                )
+            else:
+                response = self._client.filter_log_events(
+                    logGroupName=log_group_name,
+                    filterPattern=my_filter,
+                    startTime=beginning,
+                    endTime=now
+                )
+            log_events = response.get('events', [])
+            all_log_events.extend(log_events)
+            if len(log_events) > 0:
+                # We found it
+                break
+            next_token = response.get('nextToken')
+            if not next_token:
+                break
+        return all_log_events
+
+    def check_sms(self, message_id, notification_id, created_at):
+
+        # TODO this clumsy approach to getting the account number will be fixed as part of notify-api #258
+        account_number = cloud_config.ses_domain_arn
+        account_number = account_number.replace('arn:aws:ses:us-west-2:', '')
+        account_number = account_number.split(":")
+        account_number = account_number[0]
+
+        log_group_name = f'sns/us-west-2/{account_number}/DirectPublishToPhoneNumber'
+        filter_pattern = '{$.notification.messageId="XXXXX"}'
+        filter_pattern = filter_pattern.replace("XXXXX", message_id)
+        all_log_events = self._get_log(filter_pattern, log_group_name, created_at)
+
+        if all_log_events and len(all_log_events) > 0:
+            event = all_log_events[0]
+            message = json.loads(event['message'])
+            return "success", message['delivery']['providerResponse']
+
+        log_group_name = f'sns/us-west-2/{account_number}/DirectPublishToPhoneNumber/Failure'
+        all_failed_events = self._get_log(filter_pattern, log_group_name, created_at)
+        if all_failed_events and len(all_failed_events) > 0:
+            event = all_failed_events[0]
+            message = json.loads(event['message'])
+            return "failure", message['delivery']['providerResponse']
+
+        raise Exception(f'No event found for message_id {message_id} notification_id {notification_id}')

app/clients/email/aws_ses.py

@@ -1,7 +1,7 @@
 from time import monotonic
 
-import boto3
 import botocore
+from boto3 import client
 from flask import current_app
 
 from app.clients import STATISTICS_DELIVERED, STATISTICS_FAILURE
@@ -10,6 +10,7 @@ from app.clients.email import (
     EmailClientException,
     EmailClientNonRetryableException,
 )
+from app.cloudfoundry_config import cloud_config
 
 ses_response_map = {
     'Permanent': {
@@ -56,10 +57,14 @@ class AwsSesClient(EmailClient):
     Amazon SES email client.
     '''
 
-    def init_app(self, region, statsd_client, *args, **kwargs):
-        self._client = boto3.client('ses', region_name=region)
+    def init_app(self, *args, **kwargs):
+        self._client = client(
+            'ses',
+            region_name=cloud_config.ses_region,
+            aws_access_key_id=cloud_config.ses_access_key,
+            aws_secret_access_key=cloud_config.ses_secret_key
+        )
         super(AwsSesClient, self).__init__(*args, **kwargs)
-        self.statsd_client = statsd_client
 
     @property
     def name(self):
@@ -104,7 +109,6 @@ class AwsSesClient(EmailClient):
                 ReplyToAddresses=[punycode_encode_email(addr) for addr in reply_to_addresses]
             )
         except botocore.exceptions.ClientError as e:
-            self.statsd_client.incr("clients.ses.error")
 
             # http://docs.aws.amazon.com/ses/latest/DeveloperGuide/api-error-codes.html
             if e.response['Error']['Code'] == 'InvalidParameterValue':
@@ -115,16 +119,12 @@ class AwsSesClient(EmailClient):
             ):
                 raise AwsSesClientThrottlingSendRateException(str(e))
             else:
-                self.statsd_client.incr("clients.ses.error")
                 raise AwsSesClientException(str(e))
         except Exception as e:
-            self.statsd_client.incr("clients.ses.error")
             raise AwsSesClientException(str(e))
         else:
             elapsed_time = monotonic() - start_time
             current_app.logger.info("AWS SES request finished in {}".format(elapsed_time))
-            self.statsd_client.timing("clients.ses.request-time", elapsed_time)
-            self.statsd_client.incr("clients.ses.success")
             return response['MessageId']
 
 

app/clients/email/aws_ses_stub.py

@@ -12,8 +12,7 @@ class AwsSesStubClientException(EmailClientException):
 
 
 class AwsSesStubClient(EmailClient):
-    def init_app(self, region, statsd_client, stub_url):
-        self.statsd_client = statsd_client
+    def init_app(self, stub_url):
         self.url = stub_url
 
     @property
@@ -39,11 +38,8 @@ class AwsSesStubClient(EmailClient):
             response_json = json.loads(response.text)
 
         except Exception as e:
-            self.statsd_client.incr("clients.ses_stub.error")
             raise AwsSesStubClientException(str(e))
         else:
             elapsed_time = monotonic() - start_time
             current_app.logger.info("AWS SES stub request finished in {}".format(elapsed_time))
-            self.statsd_client.timing("clients.ses_stub.request-time", elapsed_time)
-            self.statsd_client.incr("clients.ses_stub.success")
             return response_json['MessageId']

app/clients/performance_platform/performance_platform_client.py

@@ -3,7 +3,6 @@ import json
 
 import requests
 from flask import current_app
-from notifications_utils.timezones import convert_utc_to_bst
 
 
 class PerformancePlatformClient:
@@ -55,7 +54,7 @@ class PerformancePlatformClient:
         :param period - the period that this data covers - "day", "week", "month", "quarter".
         """
         payload = {
-            '_timestamp': convert_utc_to_bst(start_time).isoformat(),
+            '_timestamp': start_time,
             'service': 'govuk-notify',
             'dataType': dataset,
             'period': period,

app/clients/sms/__init__.py

@@ -1,71 +1,28 @@
-from time import monotonic
-
 from app.clients import Client, ClientException
 
 
 class SmsClientResponseException(ClientException):
-    '''
+    """
     Base Exception for SmsClientsResponses
-    '''
+    """
 
     def __init__(self, message):
         self.message = message
 
     def __str__(self):
-        return f"SMS client error ({self.message})"
+        return "Message {}".format(self.message)
 
 
 class SmsClient(Client):
-    '''
+    """
     Base Sms client for sending smss.
-    '''
+    """
 
-    def init_app(self, current_app, statsd_client):
-        self.current_app = current_app
-        self.statsd_client = statsd_client
-        self.from_number = self.current_app.config.get('FROM_NUMBER')
+    def init_app(self, *args, **kwargs):
+        raise NotImplementedError("TODO Need to implement.")
 
-    def record_outcome(self, success):
-        log_message = "Provider request for {} {}".format(
-            self.name,
-            "succeeded" if success else "failed",
-        )
+    def send_sms(self, *args, **kwargs):
+        raise NotImplementedError("TODO Need to implement.")
 
-        if success:
-            self.current_app.logger.info(log_message)
-            self.statsd_client.incr(f"clients.{self.name}.success")
-        else:
-            self.statsd_client.incr(f"clients.{self.name}.error")
-            self.current_app.logger.warning(log_message)
-
-    def send_sms(self, to, content, reference, international, sender):
-        start_time = monotonic()
-
-        if sender is None:
-            # temporary log to see if the following ternary is necessary
-            # or if it's safe to remove it - keep for 1-2 weeks
-            self.current_app.logger.warning(
-                f"send_sms called with 'sender' of 'None' for {reference}"
-            )
-
-        sender = self.from_number if sender is None else sender
-
-        try:
-            response = self.try_send_sms(to, content, reference, international, sender)
-            self.record_outcome(True)
-        except SmsClientResponseException as e:
-            self.record_outcome(False)
-            raise e
-        finally:
-            elapsed_time = monotonic() - start_time
-            self.statsd_client.timing(f"clients.{self.name}.request-time", elapsed_time)
-            self.current_app.logger.info(f"{self.name} request for {reference} finished in {elapsed_time}")
-
-        return response
-
-    def try_send_sms(self, *args, **kwargs):
-        raise NotImplementedError('TODO Need to implement.')
-
-    @property
-    def name(self):
-        raise NotImplementedError('TODO Need to implement.')
+    def get_name(self):
+        raise NotImplementedError("TODO Need to implement.")

app/clients/sms/aws_sns.py

@@ -0,0 +1,79 @@
+import re
+from time import monotonic
+
+import botocore
+import phonenumbers
+from boto3 import client
+
+from app.clients.sms import SmsClient
+from app.cloudfoundry_config import cloud_config
+
+
+class AwsSnsClient(SmsClient):
+    """
+    AwsSns sms client
+    """
+
+    def init_app(self, current_app, *args, **kwargs):
+        self._client = client(
+            "sns",
+            region_name=cloud_config.sns_region,
+            aws_access_key_id=cloud_config.sns_access_key,
+            aws_secret_access_key=cloud_config.sns_secret_key
+        )
+        super(SmsClient, self).__init__(*args, **kwargs)
+        self.current_app = current_app
+        self._valid_sender_regex = re.compile(r"^\+?\d{5,14}$")
+
+    @property
+    def name(self):
+        return 'sns'
+
+    def get_name(self):
+        return self.name
+
+    def _valid_sender_number(self, sender):
+        return sender and re.match(self._valid_sender_regex, sender)
+
+    def send_sms(self, to, content, reference, sender=None, international=False):
+        matched = False
+
+        for match in phonenumbers.PhoneNumberMatcher(to, "US"):
+            matched = True
+            to = phonenumbers.format_number(match.number, phonenumbers.PhoneNumberFormat.E164)
+
+            # See documentation
+            # https://docs.aws.amazon.com/sns/latest/dg/sms_publish-to-phone.html#sms_publish_sdk
+            attributes = {
+                "AWS.SNS.SMS.SMSType": {
+                    "DataType": "String",
+                    "StringValue": "Transactional",
+                }
+            }
+
+            if self._valid_sender_number(sender):
+                attributes["AWS.MM.SMS.OriginationNumber"] = {
+                    "DataType": "String",
+                    "StringValue": sender,
+                }
+            else:
+                attributes["AWS.MM.SMS.OriginationNumber"] = {
+                    "DataType": "String",
+                    "StringValue": self.current_app.config["AWS_US_TOLL_FREE_NUMBER"],
+                }
+
+            try:
+                start_time = monotonic()
+                response = self._client.publish(PhoneNumber=to, Message=content, MessageAttributes=attributes)
+            except botocore.exceptions.ClientError as e:
+                raise str(e)
+            except Exception as e:
+                raise str(e)
+            finally:
+                elapsed_time = monotonic() - start_time
+                self.current_app.logger.info("AWS SNS request finished in {}".format(elapsed_time))
+            return response["MessageId"]
+
+        if not matched:
+            self.current_app.logger.error("No valid numbers found in {}".format(to))
+            raise ValueError("No valid numbers found for SMS delivery")

app/clients/sms/firetext.py

@@ -1,88 +0,0 @@
-import json
-import logging
-
-from requests import RequestException, request
-
-from app.clients.sms import SmsClient, SmsClientResponseException
-
-logger = logging.getLogger(__name__)
-
-# Firetext will send a delivery receipt with three different status codes.
-# The `firetext_response` maps these codes to the notification statistics status and notification status.
-# If we get a pending (status = 2) delivery receipt followed by a declined (status = 1) delivery receipt we will set
-# the notification status to temporary-failure rather than permanent failure.
-#  See the code in the notification_dao.update_notifications_status_by_id
-firetext_responses = {
-    '0': 'delivered',
-    '1': 'permanent-failure',
-    '2': 'pending'
-}
-
-firetext_codes = {
-    # code '000' means 'No errors reported'
-    '000': {'status': 'temporary-failure', 'reason': 'No error reported'},
-    '101': {'status': 'permanent-failure', 'reason': 'Unknown Subscriber'},
-    '102': {'status': 'temporary-failure', 'reason': 'Absent Subscriber'},
-    '103': {'status': 'temporary-failure', 'reason': 'Subscriber Busy'},
-    '104': {'status': 'temporary-failure', 'reason': 'No Subscriber Memory'},
-    '201': {'status': 'permanent-failure', 'reason': 'Invalid Number'},
-    '301': {'status': 'permanent-failure', 'reason': 'SMS Not Supported'},
-    '302': {'status': 'temporary-failure', 'reason': 'SMS Not Supported'},
-    '401': {'status': 'permanent-failure', 'reason': 'Message Rejected'},
-    '900': {'status': 'temporary-failure', 'reason': 'Routing Error'},
-}
-
-
-def get_firetext_responses(status, detailed_status_code=None):
-    detailed_status = firetext_codes[detailed_status_code]['reason'] if firetext_codes.get(
-        detailed_status_code, None
-    ) else None
-    return (firetext_responses[status], detailed_status)
-
-
-def get_message_status_and_reason_from_firetext_code(detailed_status_code):
-    return firetext_codes[detailed_status_code]['status'], firetext_codes[detailed_status_code]['reason']
-
-
-class FiretextClient(SmsClient):
-    '''
-    FireText sms client.
-    '''
-
-    def init_app(self, *args, **kwargs):
-        super().init_app(*args, **kwargs)
-        self.api_key = self.current_app.config.get('FIRETEXT_API_KEY')
-        self.international_api_key = self.current_app.config.get('FIRETEXT_INTERNATIONAL_API_KEY')
-        self.url = self.current_app.config.get('FIRETEXT_URL')
-
-    @property
-    def name(self):
-        return 'firetext'
-
-    def try_send_sms(self, to, content, reference, international, sender):
-        data = {
-            "apiKey": self.international_api_key if international else self.api_key,
-            "from": sender,
-            "to": to.replace('+', ''),
-            "message": content,
-            "reference": reference
-        }
-
-        try:
-            response = request(
-                "POST",
-                self.url,
-                data=data,
-                timeout=60
-            )
-            response.raise_for_status()
-            try:
-                json.loads(response.text)
-                if response.json()['code'] != 0:
-                    raise ValueError("Expected 'code' to be '0'")
-            except (ValueError, AttributeError):
-                raise SmsClientResponseException("Invalid response JSON")
-        except RequestException:
-            raise SmsClientResponseException("Request failed")
-
-        return response

app/clients/sms/mmg.py

@@ -1,110 +0,0 @@
-import json
-
-from requests import RequestException, request
-
-from app.clients.sms import SmsClient, SmsClientResponseException
-
-mmg_response_map = {
-    '2': {'status': 'permanent-failure', 'substatus': {
-        "1": "Number does not exist",
-        "4": "Rejected by operator",
-        "5": "Unidentified Subscriber",
-        "9": "Undelivered",
-        "11": "Service for Subscriber suspended",
-        "12": "Illegal equipment",
-        "2049": "Subscriber IMSI blacklisted",
-        "2050": "Number blacklisted in do-not-disturb blacklist",
-        "2052": "Destination number blacklisted",
-        "2053": "Source address blacklisted"
-    }},
-    '3': {'status': 'delivered', 'substatus': {"2": "Delivered to operator", "5": "Delivered to handset"}},
-    '4': {'status': 'temporary-failure', 'substatus': {
-        "6": "Absent Subscriber",
-        "8": "Roaming not allowed",
-        "13": "SMS Not Supported",
-        "15": "Expired",
-        "27": "Absent Subscriber",
-        "29": "Invalid delivery report",
-        "32": "Delivery Failure",
-    }},
-    '5': {'status': 'permanent-failure', 'substatus': {
-        "6": "Network out of coverage",
-        "8": "Incorrect number prefix",
-        "10": "Number on do-not-disturb service",
-        "11": "Sender id not registered",
-        "13": "Sender id blacklisted",
-        "14": "Destination number blacklisted",
-        "19": "Routing unavailable",
-        "20": "Rejected by anti-flooding mechanism",
-        "21": "System error",  # it says to retry those messages or contact support
-        "23": "Duplicate message id",
-        "24": "Message formatted incorrectly",
-        "25": "Message too long",
-        "51": "Missing recipient value",
-        "52": "Invalid destination",
-    }},
-}
-
-
-def get_mmg_responses(status, detailed_status_code=None):
-    return (mmg_response_map[status]["status"], mmg_response_map[status]["substatus"].get(detailed_status_code, None))
-
-
-class MMGClientResponseException(SmsClientResponseException):
-    def __init__(self, response, exception):
-        status_code = response.status_code if response is not None else 504
-        text = response.text if response is not None else "Gateway Time-out"
-
-        self.status_code = status_code
-        self.text = text
-        self.exception = exception
-
-    def __str__(self):
-        return "Code {} text {} exception {}".format(self.status_code, self.text, str(self.exception))
-
-
-class MMGClient(SmsClient):
-    '''
-    MMG sms client
-    '''
-
-    def init_app(self, *args, **kwargs):
-        super().init_app(*args, **kwargs)
-        self.api_key = self.current_app.config.get('MMG_API_KEY')
-        self.mmg_url = self.current_app.config.get('MMG_URL')
-
-    @property
-    def name(self):
-        return 'mmg'
-
-    def try_send_sms(self, to, content, reference, international, sender):
-        data = {
-            "reqType": "BULK",
-            "MSISDN": to,
-            "msg": content,
-            "sender": sender,
-            "cid": reference,
-            "multi": True
-        }
-
-        try:
-            response = request(
-                "POST",
-                self.mmg_url,
-                data=json.dumps(data),
-                headers={
-                    'Content-Type': 'application/json',
-                    'Authorization': 'Basic {}'.format(self.api_key)
-                },
-                timeout=60
-            )
-
-            response.raise_for_status()
-            try:
-                json.loads(response.text)
-            except (ValueError, AttributeError):
-                raise SmsClientResponseException("Invalid response JSON")
-        except RequestException:
-            raise SmsClientResponseException("Request failed")
-
-        return response

app/clients/sms/reach.py

@@ -1,52 +0,0 @@
-import json
-
-from requests import RequestException, request
-
-from app.clients.sms import SmsClient, SmsClientResponseException
-
-
-def get_reach_responses(status, detailed_status_code=None):
-    if status == 'TODO-d':
-        return ("delivered", "TODO: Delivered")
-    elif status == 'TODO-tf':
-        return ("temporary-failure", "TODO: Temporary failure")
-    elif status == 'TODO-pf':
-        return ("permanent-failure", "TODO: Permanent failure")
-    else:
-        raise KeyError
-
-
-class ReachClient(SmsClient):
-    def init_app(self, *args, **kwargs):
-        super().init_app(*args, **kwargs)
-        self.url = self.current_app.config.get('REACH_URL')
-
-    @property
-    def name(self):
-        return 'reach'
-
-    def try_send_sms(self, to, content, reference, international, sender):
-        data = {
-            # TODO
-        }
-
-        try:
-            response = request(
-                "POST",
-                self.url,
-                data=json.dumps(data),
-                headers={
-                    'Content-Type': 'application/json',
-                },
-                timeout=60
-            )
-
-            response.raise_for_status()
-            try:
-                json.loads(response.text)
-            except (ValueError, AttributeError):
-                raise SmsClientResponseException("Invalid response JSON")
-        except RequestException:
-            raise SmsClientResponseException("Request failed")
-
-        return response

app/cloudfoundry_config.py

@@ -1,22 +1,111 @@
-"""
-Extracts cloudfoundry config from its json and populates the environment variables that we would expect to be populated
-on local/aws boxes
-"""
-
 import json
-import os
+from os import getenv
 
 
-def extract_cloudfoundry_config():
-    vcap_services = json.loads(os.environ['VCAP_SERVICES'])
-    set_config_env_vars(vcap_services)
+class CloudfoundryConfig:
+    def __init__(self):
+        self.parsed_services = json.loads(getenv('VCAP_SERVICES') or '{}')
+        buckets = self.parsed_services.get('s3') or []
+        self.s3_buckets = {bucket['name']: bucket['credentials'] for bucket in buckets}
+        self._empty_bucket_credentials = {
+            'bucket': '',
+            'access_key_id': '',
+            'secret_access_key': '',
+            'region': ''
+        }
+
+    @property
+    def database_url(self):
+        return getenv('DATABASE_URL', '').replace('postgres://', 'postgresql://')
+
+    @property
+    def redis_url(self):
+        try:
+            return self.parsed_services['aws-elasticache-redis'][0]['credentials']['uri'].replace(
+                'redis://',
+                'rediss://'
+            )
+        except KeyError:
+            return getenv('REDIS_URL')
+
+    def s3_credentials(self, service_name):
+        return self.s3_buckets.get(service_name) or self._empty_bucket_credentials
+
+    @property
+    def ses_email_domain(self):
+        try:
+            domain_arn = self._ses_credentials('domain_arn')
+        except KeyError:
+            domain_arn = getenv('SES_DOMAIN_ARN', 'dev.notify.gov')
+        return domain_arn.split('/')[-1]
+
+    # TODO remove this after notifications-api #258
+    @property
+    def ses_domain_arn(self):
+        try:
+            domain_arn = self._ses_credentials('domain_arn')
+        except KeyError:
+            domain_arn = getenv('SES_DOMAIN_ARN', 'dev.notify.gov')
+        return domain_arn
+
+    @property
+    def ses_region(self):
+        try:
+            return self._ses_credentials('region')
+        except KeyError:
+            return getenv('SES_AWS_REGION', 'us-west-1')
+
+    @property
+    def ses_access_key(self):
+        try:
+            return self._ses_credentials('smtp_user')
+        except KeyError:
+            return getenv('SES_AWS_ACCESS_KEY_ID')
+
+    @property
+    def ses_secret_key(self):
+        try:
+            return self._ses_credentials('secret_access_key')
+        except KeyError:
+            return getenv('SES_AWS_SECRET_ACCESS_KEY')
+
+    @property
+    def sns_access_key(self):
+        try:
+            return self._sns_credentials('aws_access_key_id')
+        except KeyError:
+            return getenv('SNS_AWS_ACCESS_KEY_ID')
+
+    @property
+    def sns_secret_key(self):
+        try:
+            return self._sns_credentials('aws_secret_access_key')
+        except KeyError:
+            return getenv('SNS_AWS_SECRET_ACCESS_KEY')
+
+    @property
+    def sns_region(self):
+        try:
+            return self._sns_credentials('region')
+        except KeyError:
+            return getenv('SNS_AWS_REGION', 'us-west-1')
+
+    @property
+    def sns_topic_arns(self):
+        try:
+            return [
+                self._ses_credentials('bounce_topic_arn'),
+                self._ses_credentials('complaint_topic_arn'),
+                self._ses_credentials('delivery_topic_arn')
+            ]
+        except KeyError:
+            return []
+
+    def _ses_credentials(self, key):
+        return self.parsed_services['datagov-smtp'][0]['credentials'][key]
+
+    def _sns_credentials(self, key):
+        return self.parsed_services['ttsnotify-sms'][0]['credentials'][key]
 
 
-def set_config_env_vars(vcap_services):
-    # Postgres config
-    os.environ['SQLALCHEMY_DATABASE_URI'] = vcap_services['postgres'][0]['credentials']['uri'].replace('postgres',
-                                                                                                       'postgresql')
-
-    vcap_application = json.loads(os.environ['VCAP_APPLICATION'])
-    os.environ['NOTIFY_ENVIRONMENT'] = vcap_application['space_name']
-    os.environ['NOTIFY_LOG_PATH'] = '/home/vcap/logs/app.log'
+cloud_config = CloudfoundryConfig()

app/commands.py

@@ -1,17 +1,16 @@
 import csv
 import functools
 import itertools
-import os
 import uuid
 from datetime import datetime, timedelta
-from decimal import Decimal
+from os import getenv
 
 import click
 import flask
 from click_datetime import Datetime as click_dt
 from flask import current_app, json
+from notifications_python_client.authentication import create_jwt_token
 from notifications_utils.recipients import RecipientCSV
-from notifications_utils.statsd_decorators import statsd
 from notifications_utils.template import SMSMessageTemplate
 from sqlalchemy import and_
 from sqlalchemy.exc import IntegrityError
@@ -19,12 +18,8 @@ from sqlalchemy.orm.exc import NoResultFound
 
 from app import db
 from app.aws import s3
-from app.celery.letters_pdf_tasks import (
-    get_pdf_for_templated_letter,
-    resanitise_pdf,
-)
-from app.celery.tasks import process_row, record_daily_sorted_counts
-from app.config import QueueNames
+from app.celery.nightly_tasks import cleanup_unfinished_jobs
+from app.celery.tasks import process_row
 from app.dao.annual_billing_dao import (
     dao_create_or_update_annual_billing_for_year,
     set_default_free_allowance_for_service,
@@ -36,14 +31,10 @@ from app.dao.fact_billing_dao import (
     update_fact_billing,
 )
 from app.dao.jobs_dao import dao_get_job_by_id
-from app.dao.organisation_dao import (
-    dao_add_service_to_organisation,
-    dao_get_organisation_by_email_address,
-    dao_get_organisation_by_id,
-)
-from app.dao.permissions_dao import permission_dao
-from app.dao.provider_rates_dao import (
-    create_provider_rates as dao_create_provider_rates,
+from app.dao.organization_dao import (
+    dao_add_service_to_organization,
+    dao_get_organization_by_email_address,
+    dao_get_organization_by_id,
 )
 from app.dao.services_dao import (
     dao_fetch_all_services_by_user,
@@ -61,19 +52,18 @@ from app.dao.users_dao import (
 from app.models import (
     KEY_TYPE_TEST,
     NOTIFICATION_CREATED,
-    PROVIDERS,
     SMS_TYPE,
     AnnualBilling,
     Domain,
     EmailBranding,
-    LetterBranding,
     Notification,
-    Organisation,
-    Permission,
+    Organization,
     Service,
+    Template,
+    TemplateHistory,
     User,
 )
-from app.utils import get_london_midnight_in_utc
+from app.utils import get_midnight_in_utc
 
 
 @click.group(name='command', help='Additional commands')
@@ -93,7 +83,7 @@ class notify_command:
 
         # in the test environment the app context is already provided and having
         # another will lead to the test db connection being closed prematurely
-        if os.getenv('NOTIFY_ENVIRONMENT', '') != 'test':
+        if getenv('NOTIFY_ENVIRONMENT', '') != 'test':
             # with_appcontext ensures the config is loaded, db connected, etc.
             decorators.insert(0, flask.cli.with_appcontext)
 
@@ -108,18 +98,6 @@ class notify_command:
         return wrapper
 
 
-@notify_command()
-@click.option('-p', '--provider_name', required=True, type=click.Choice(PROVIDERS))
-@click.option('-c', '--cost', required=True, help='Cost (pence) per message including decimals', type=float)
-@click.option('-d', '--valid_from', required=True, type=click_dt(format='%Y-%m-%dT%H:%M:%S'))
-def create_provider_rates(provider_name, cost, valid_from):
-    """
-    Backfill rates for a given provider
-    """
-    cost = Decimal(cost)
-    dao_create_provider_rates(provider_name, valid_from, cost)
-
-
 @notify_command()
 @click.option('-u', '--user_email_prefix', required=True, help="""
     Functional test user email prefix. eg "notify-test-preview"
@@ -130,6 +108,10 @@ def purge_functional_test_data(user_email_prefix):
 
     users, services, etc. Give an email prefix. Probably "notify-tests-preview".
     """
+    if getenv('NOTIFY_ENVIRONMENT', '') not in ['development', 'test']:
+        current_app.logger.error('Can only be run in development')
+        return
+
     users = User.query.filter(User.email_address.like("{}%".format(user_email_prefix))).all()
     for usr in users:
         # Make sure the full email includes a uuid in it
@@ -159,95 +141,13 @@ def purge_functional_test_data(user_email_prefix):
                 delete_model_user(usr)
 
 
-@notify_command()
-def backfill_notification_statuses():
-    """
-    DEPRECATED. Populates notification_status.
-
-    This will be used to populate the new `Notification._status_fkey` with the old
-    `Notification._status_enum`
-    """
-    LIMIT = 250000
-    subq = "SELECT id FROM notification_history WHERE notification_status is NULL LIMIT {}".format(LIMIT)
-    update = "UPDATE notification_history SET notification_status = status WHERE id in ({})".format(subq)
-    result = db.session.execute(subq).fetchall()
-
-    while len(result) > 0:
-        db.session.execute(update)
-        print('commit {} updates at {}'.format(LIMIT, datetime.utcnow()))
-        db.session.commit()
-        result = db.session.execute(subq).fetchall()
-
-
-@notify_command()
-def update_notification_international_flag():
-    """
-    DEPRECATED. Set notifications.international=false.
-    """
-    # 250,000 rows takes 30 seconds to update.
-    subq = "select id from notifications where international is null limit 250000"
-    update = "update notifications set international = False where id in ({})".format(subq)
-    result = db.session.execute(subq).fetchall()
-
-    while len(result) > 0:
-        db.session.execute(update)
-        print('commit 250000 updates at {}'.format(datetime.utcnow()))
-        db.session.commit()
-        result = db.session.execute(subq).fetchall()
-
-    # Now update notification_history
-    subq_history = "select id from notification_history where international is null limit 250000"
-    update_history = "update notification_history set international = False where id in ({})".format(subq_history)
-    result_history = db.session.execute(subq_history).fetchall()
-    while len(result_history) > 0:
-        db.session.execute(update_history)
-        print('commit 250000 updates at {}'.format(datetime.utcnow()))
-        db.session.commit()
-        result_history = db.session.execute(subq_history).fetchall()
-
-
-@notify_command()
-def fix_notification_statuses_not_in_sync():
-    """
-    DEPRECATED.
-    This will be used to correct an issue where Notification._status_enum and NotificationHistory._status_fkey
-    became out of sync. See 979e90a.
-
-    Notification._status_enum is the source of truth so NotificationHistory._status_fkey will be updated with
-    these values.
-    """
-    MAX = 10000
-
-    subq = "SELECT id FROM notifications WHERE cast (status as text) != notification_status LIMIT {}".format(MAX)
-    update = "UPDATE notifications SET notification_status = status WHERE id in ({})".format(subq)
-    result = db.session.execute(subq).fetchall()
-
-    while len(result) > 0:
-        db.session.execute(update)
-        print('Committed {} updates at {}'.format(len(result), datetime.utcnow()))
-        db.session.commit()
-        result = db.session.execute(subq).fetchall()
-
-    subq_hist = "SELECT id FROM notification_history WHERE cast (status as text) != notification_status LIMIT {}" \
-        .format(MAX)
-    update = "UPDATE notification_history SET notification_status = status WHERE id in ({})".format(subq_hist)
-    result = db.session.execute(subq_hist).fetchall()
-
-    while len(result) > 0:
-        db.session.execute(update)
-        print('Committed {} updates at {}'.format(len(result), datetime.utcnow()))
-        db.session.commit()
-        result = db.session.execute(subq_hist).fetchall()
-
-
 @notify_command(name='insert-inbound-numbers')
 @click.option('-f', '--file_name', required=True,
-              help="""Full path of the file to upload, file is a contains inbound numbers,
-              one number per line. The number must have the format of 07... not 447....""")
+              help="""Full path of the file to upload, file is a contains inbound numbers, one number per line.""")
 def insert_inbound_numbers_from_file(file_name):
     print("Inserting inbound numbers from {}".format(file_name))
     with open(file_name) as file:
-        sql = "insert into inbound_numbers values('{}', '{}', 'mmg', null, True, now(), null);"
+        sql = "insert into inbound_numbers values('{}', '{}', 'sns', null, True, now(), null);"
 
         for line in file:
             line = line.strip()
@@ -257,22 +157,6 @@ def insert_inbound_numbers_from_file(file_name):
                 db.session.commit()
 
 
-@notify_command(name='replay-create-pdf-for-templated-letter')
-@click.option('-n', '--notification_id', type=click.UUID, required=True,
-              help="Notification id of the letter that needs the get_pdf_for_templated_letter task replayed")
-def replay_create_pdf_for_templated_letter(notification_id):
-    print("Create task to get_pdf_for_templated_letter for notification: {}".format(notification_id))
-    get_pdf_for_templated_letter.apply_async([str(notification_id)], queue=QueueNames.CREATE_LETTERS_PDF)
-
-
-@notify_command(name='recreate-pdf-for-precompiled-or-uploaded-letter')
-@click.option('-n', '--notification_id', type=click.UUID, required=True,
-              help="Notification ID of the precompiled or uploaded letter")
-def recreate_pdf_for_precompiled_or_uploaded_letter(notification_id):
-    print(f"Call resanitise_pdf task for notification: {notification_id}")
-    resanitise_pdf.apply_async([str(notification_id)], queue=QueueNames.LETTERS)
-
-
 def setup_commands(application):
     application.cli.add_command(command_group)
 
@@ -309,8 +193,8 @@ def rebuild_ft_billing_for_day(service_id, day):
         rebuild_ft_data(day, service_id)
     else:
         services = get_service_ids_that_need_billing_populated(
-            get_london_midnight_in_utc(day),
-            get_london_midnight_in_utc(day + timedelta(days=1))
+            get_midnight_in_utc(day),
+            get_midnight_in_utc(day + timedelta(days=1))
         )
         for row in services:
             rebuild_ft_data(day, row.service_id)
@@ -327,11 +211,11 @@ def rebuild_ft_billing_for_day(service_id, day):
 def bulk_invite_user_to_service(file_name, service_id, user_id, auth_type, permissions):
     #  permissions
     #  manage_users | manage_templates | manage_settings
-    #  send messages ==> send_texts | send_emails | send_letters
+    #  send messages ==> send_texts | send_emails
     #  Access API keys manage_api_keys
     #  platform_admin
     #  view_activity
-    # "send_texts,send_emails,send_letters,view_activity"
+    # "send_texts,send_emails,view_activity"
     from app.service_invite.rest import create_invited_user
     file = open(file_name)
     for email_address in file:
@@ -360,56 +244,9 @@ def bulk_invite_user_to_service(file_name, service_id, user_id, auth_type, permi
     file.close()
 
 
-@notify_command(name='populate-notification-postage')
-@click.option(
-    '-s',
-    '--start_date',
-    default=datetime(2017, 2, 1),
-    help="start date inclusive",
-    type=click_dt(format='%Y-%m-%d')
-)
-@statsd(namespace="tasks")
-def populate_notification_postage(start_date):
-    current_app.logger.info('populating historical notification postage')
-
-    total_updated = 0
-
-    while start_date < datetime.utcnow():
-        # process in ten day chunks
-        end_date = start_date + timedelta(days=10)
-
-        sql = \
-            """
-            UPDATE {}
-            SET postage = 'second'
-            WHERE notification_type = 'letter' AND
-            postage IS NULL AND
-            created_at BETWEEN :start AND :end
-            """
-
-        execution_start = datetime.utcnow()
-
-        if end_date > datetime.utcnow() - timedelta(days=8):
-            print('Updating notifications table as well')
-            db.session.execute(sql.format('notifications'), {'start': start_date, 'end': end_date})
-
-        result = db.session.execute(sql.format('notification_history'), {'start': start_date, 'end': end_date})
-        db.session.commit()
-
-        current_app.logger.info('notification postage took {}ms. Migrated {} rows for {} to {}'.format(
-            datetime.utcnow() - execution_start, result.rowcount, start_date, end_date))
-
-        start_date += timedelta(days=10)
-
-        total_updated += result.rowcount
-
-    current_app.logger.info('Total inserted/updated records = {}'.format(total_updated))
-
-
 @notify_command(name='archive-jobs-created-between-dates')
 @click.option('-s', '--start_date', required=True, help="start date inclusive", type=click_dt(format='%Y-%m-%d'))
 @click.option('-e', '--end_date', required=True, help="end date inclusive", type=click_dt(format='%Y-%m-%d'))
-@statsd(namespace="tasks")
 def update_jobs_archived_flag(start_date, end_date):
     current_app.logger.info('Archiving jobs created between {} to {}'.format(start_date, end_date))
 
@@ -421,9 +258,9 @@ def update_jobs_archived_flag(start_date, end_date):
         sql = """update
                     jobs set archived = true
                 where
-                    created_at >= (date :start + time '00:00:00') at time zone 'Europe/London'
+                    created_at >= (date :start + time '00:00:00') at time zone 'America/New_York'
                     at time zone 'UTC'
-                    and created_at < (date :end + time '00:00:00') at time zone 'Europe/London' at time zone 'UTC'"""
+                    and created_at < (date :end + time '00:00:00') at time zone 'America/New_York' at time zone 'UTC'"""
 
         result = db.session.execute(sql, {"start": process_date, "end": process_date + timedelta(days=1)})
         db.session.commit()
@@ -436,59 +273,18 @@ def update_jobs_archived_flag(start_date, end_date):
     current_app.logger.info('Total archived jobs = {}'.format(total_updated))
 
 
-@notify_command(name='update-emails-to-remove-gsi')
-@click.option('-s', '--service_id', required=True, help="service id. Update all user.email_address to remove .gsi")
-@statsd(namespace="tasks")
-def update_emails_to_remove_gsi(service_id):
-    users_to_update = """SELECT u.id user_id, u.name, email_address, s.id, s.name
-                           FROM users u
-                           JOIN user_to_service us on (u.id = us.user_id)
-                           JOIN services s on (s.id = us.service_id)
-                          WHERE s.id = :service_id
-                            AND u.email_address ilike ('%.gsi.gov.uk%')
-    """
-    results = db.session.execute(users_to_update, {'service_id': service_id})
-    print("Updating {} users.".format(results.rowcount))
-
-    for user in results:
-        print('User with id {} updated'.format(user.user_id))
-
-        update_stmt = """
-        UPDATE users
-           SET email_address = replace(replace(email_address, '.gsi.gov.uk', '.gov.uk'), '.GSI.GOV.UK', '.GOV.UK'),
-               updated_at = now()
-         WHERE id = :user_id
-        """
-        db.session.execute(update_stmt, {'user_id': str(user.user_id)})
-        db.session.commit()
-
-
-@notify_command(name='replay-daily-sorted-count-files')
-@click.option('-f', '--file_extension', required=False, help="File extension to search for, defaults to rs.txt")
-@statsd(namespace="tasks")
-def replay_daily_sorted_count_files(file_extension):
-    bucket_location = '{}-ftp'.format(current_app.config['NOTIFY_EMAIL_DOMAIN'])
-    for filename in s3.get_list_of_files_by_suffix(bucket_name=bucket_location,
-                                                   subfolder='root/dispatch',
-                                                   suffix=file_extension or '.rs.txt'):
-        print("Create task to record daily sorted counts for file: ", filename)
-        record_daily_sorted_counts.apply_async([filename], queue=QueueNames.NOTIFY)
-
-
-@notify_command(name='populate-organisations-from-file')
+@notify_command(name='populate-organizations-from-file')
 @click.option('-f', '--file_name', required=True,
-              help="Pipe delimited file containing organisation name, sector, crown, argeement_signed, domains")
-def populate_organisations_from_file(file_name):
-    # [0] organisation name:: name of the organisation insert if organisation is missing.
-    # [1] sector:: Central | Local | NHS only
-    # [2] crown:: TRUE | FALSE only
-    # [3] argeement_signed:: TRUE | FALSE
-    # [4] domains:: comma separated list of domains related to the organisation
-    # [5] email branding name: name of the default email branding for the org
-    # [6] letter branding name: name of the default letter branding for the org
+              help="Pipe delimited file containing organization name, sector, agreement_signed, domains")
+def populate_organizations_from_file(file_name):
+    # [0] organization name:: name of the organization insert if organization is missing.
+    # [1] sector:: Federal | State only
+    # [2] agreement_signed:: TRUE | FALSE
+    # [3] domains:: comma separated list of domains related to the organization
+    # [4] email branding name: name of the default email branding for the org
 
-    # The expectation is that the organisation, organisation_to_service
-    # and user_to_organisation will be cleared before running this command.
+    # The expectation is that the organization, organization_to_service
+    # and user_to_organization will be cleared before running this command.
     # Ignoring duplicates allows us to run the command again with the same file or same file with new rows.
     with open(file_name, 'r') as f:
         def boolean_or_none(field):
@@ -506,21 +302,14 @@ def populate_organisations_from_file(file_name):
             email_branding_column = columns[5].strip()
             if len(email_branding_column) > 0:
                 email_branding = EmailBranding.query.filter(EmailBranding.name == email_branding_column).one()
-            letter_branding = None
-            letter_branding_column = columns[6].strip()
-            if len(letter_branding_column) > 0:
-                letter_branding = LetterBranding.query.filter(LetterBranding.name == letter_branding_column).one()
             data = {
                 'name': columns[0],
                 'active': True,
                 'agreement_signed': boolean_or_none(columns[3]),
-                'crown': boolean_or_none(columns[2]),
-                'organisation_type': columns[1].lower(),
-                'email_branding_id': email_branding.id if email_branding else None,
-                'letter_branding_id': letter_branding.id if letter_branding else None
-
+                'organization_type': columns[1].lower(),
+                'email_branding_id': email_branding.id if email_branding else None
             }
-            org = Organisation(**data)
+            org = Organization(**data)
             try:
                 db.session.add(org)
                 db.session.commit()
@@ -530,7 +319,7 @@ def populate_organisations_from_file(file_name):
             domains = columns[4].split(',')
             for d in domains:
                 if len(d.strip()) > 0:
-                    domain = Domain(domain=d.strip(), organisation_id=org.id)
+                    domain = Domain(domain=d.strip(), organization_id=org.id)
                     try:
                         db.session.add(domain)
                         db.session.commit()
@@ -539,14 +328,14 @@ def populate_organisations_from_file(file_name):
                         db.session.rollback()
 
 
-@notify_command(name='populate-organisation-agreement-details-from-file')
+@notify_command(name='populate-organization-agreement-details-from-file')
 @click.option('-f', '--file_name', required=True,
               help="CSV file containing id, agreement_signed_version, "
               "agreement_signed_on_behalf_of_name, agreement_signed_at")
-def populate_organisation_agreement_details_from_file(file_name):
+def populate_organization_agreement_details_from_file(file_name):
     """
     The input file should be a comma separated CSV file with a header row and 4 columns
-    id: the organisation ID
+    id: the organization ID
     agreement_signed_version
     agreement_signed_on_behalf_of_name
     agreement_signed_at: The date the agreement was signed in the format of 'dd/mm/yyyy'
@@ -558,11 +347,12 @@ def populate_organisation_agreement_details_from_file(file_name):
         next(csv_reader)
 
         for row in csv_reader:
-            org = dao_get_organisation_by_id(row[0])
+            org = dao_get_organization_by_id(row[0])
 
             current_app.logger.info(f"Updating {org.name}")
 
-            assert org.agreement_signed
+            if not org.agreement_signed:
+                raise RuntimeError('Agreement was not signed')
 
             org.agreement_signed_version = float(row[1])
             org.agreement_signed_on_behalf_of_name = row[2].strip()
@@ -572,81 +362,29 @@ def populate_organisation_agreement_details_from_file(file_name):
             db.session.commit()
 
 
-@notify_command(name='get-letter-details-from-zips-sent-file')
-@click.argument('file_paths', required=True, nargs=-1)
-@statsd(namespace="tasks")
-def get_letter_details_from_zips_sent_file(file_paths):
-    """Get notification details from letters listed in zips_sent file(s)
-
-    This takes one or more file paths for the zips_sent files in S3 as its parameters, for example:
-    get-letter-details-from-zips-sent-file '2019-04-01/zips_sent/filename_1' '2019-04-01/zips_sent/filename_2'
-    """
-
-    rows_from_file = []
-
-    for path in file_paths:
-        file_contents = s3.get_s3_file(
-            bucket_name=current_app.config['LETTERS_PDF_BUCKET_NAME'],
-            file_location=path
-        )
-        rows_from_file.extend(json.loads(file_contents))
-
-    notification_references = tuple(row[18:34] for row in rows_from_file)
-    get_letters_data_from_references(notification_references)
-
-
-@notify_command(name='get-notification-and-service-ids-for-letters-that-failed-to-print')
-@click.option('-f', '--file_name', required=True,
-              help="""Full path of the file to upload, file should contain letter filenames, one per line""")
-def get_notification_and_service_ids_for_letters_that_failed_to_print(file_name):
-    print("Getting service and notification ids for letter filenames list {}".format(file_name))
-    file = open(file_name)
-    references = tuple([row[7:23] for row in file])
-
-    get_letters_data_from_references(tuple(references))
-    file.close()
-
-
-def get_letters_data_from_references(notification_references):
-    sql = """
-        SELECT id, service_id, template_id, reference, job_id, created_at
-        FROM notifications
-        WHERE reference IN :notification_references
-        ORDER BY service_id, job_id"""
-    result = db.session.execute(sql, {'notification_references': notification_references}).fetchall()
-
-    with open('zips_sent_details.csv', 'w') as csvfile:
-        csv_writer = csv.writer(csvfile)
-        csv_writer.writerow(['notification_id', 'service_id', 'template_id', 'reference', 'job_id', 'created_at'])
-
-        for row in result:
-            csv_writer.writerow(row)
-
-
-@notify_command(name='associate-services-to-organisations')
-def associate_services_to_organisations():
+@notify_command(name='associate-services-to-organizations')
+def associate_services_to_organizations():
     services = Service.get_history_model().query.filter_by(
         version=1
     ).all()
 
     for s in services:
         created_by_user = User.query.filter_by(id=s.created_by_id).first()
-        organisation = dao_get_organisation_by_email_address(created_by_user.email_address)
+        organization = dao_get_organization_by_email_address(created_by_user.email_address)
         service = dao_fetch_service_by_id(service_id=s.id)
-        if organisation:
-            dao_add_service_to_organisation(service=service, organisation_id=organisation.id)
+        if organization:
+            dao_add_service_to_organization(service=service, organization_id=organization.id)
 
-    print("finished associating services to organisations")
+    print("finished associating services to organizations")
 
 
 @notify_command(name='populate-service-volume-intentions')
 @click.option('-f', '--file_name', required=True,
-              help="Pipe delimited file containing service_id, SMS, email, letters")
+              help="Pipe delimited file containing service_id, SMS, email")
 def populate_service_volume_intentions(file_name):
     # [0] service_id
     # [1] SMS:: volume intentions for service
     # [2] Email:: volume intentions for service
-    # [3] Letters:: volume intentions for service
 
     with open(file_name, 'r') as f:
         for line in itertools.islice(f, 1, None):
@@ -655,7 +393,6 @@ def populate_service_volume_intentions(file_name):
             service = dao_fetch_service_by_id(columns[0])
             service.volume_sms = columns[1]
             service.volume_email = columns[2]
-            service.volume_letter = columns[3]
             dao_update_service(service)
     print("populate-service-volume-intentions complete")
 
@@ -728,6 +465,12 @@ def fix_billable_units():
     print("End fix_billable_units")
 
 
+@notify_command(name='delete-unfinished-jobs')
+def delete_unfinished_jobs():
+    cleanup_unfinished_jobs()
+    print("End cleanup_unfinished_jobs")
+
+
 @notify_command(name='process-row-from-job')
 @click.option('-j', '--job_id', required=True, help='Job id')
 @click.option('-n', '--job_row_number', type=int, required=True, help='Job id')
@@ -830,31 +573,85 @@ def populate_annual_billing_with_defaults(year, missing_services_only):
             set_default_free_allowance_for_service(service, year)
 
 
-@click.option('-u', '--user-id', required=True)
-@notify_command(name='local-dev-broadcast-permissions')
-def local_dev_broadcast_permissions(user_id):
-    if os.getenv('NOTIFY_ENVIRONMENT', '') not in ['development', 'test']:
+def validate_mobile(ctx, param, value):
+    if (len(''.join(i for i in value if i.isdigit())) != 10):
+        raise click.BadParameter("mobile number must have 10 digits")
+    else:
+        return value
+
+
+@notify_command(name='create-test-user')
+@click.option('-n', '--name', required=True, prompt=True)
+@click.option('-e', '--email', required=True, prompt=True)  # TODO: require valid email
+@click.option('-m', '--mobile_number',
+              required=True, prompt=True, callback=validate_mobile)
+@click.option('-p', '--password',
+              required=True, prompt=True, hide_input=True, confirmation_prompt=True)
+@click.option('-a', '--auth_type', default="sms_auth")
+@click.option('-s', '--state', default="active")
+@click.option('-d', '--admin', default=False, type=bool)
+def create_test_user(name, email, mobile_number, password, auth_type, state, admin):
+    if getenv('NOTIFY_ENVIRONMENT', '') not in ['development', 'test']:
         current_app.logger.error('Can only be run in development')
         return
 
-    user = User.query.filter_by(id=user_id).one()
+    data = {
+        'name': name,
+        'email_address': email,
+        'mobile_number': mobile_number,
+        'password': password,
+        'auth_type': auth_type,
+        'state': state,  # skip the email verification for our test user
+        'platform_admin': admin,
+    }
+    user = User(**data)
+    try:
+        db.session.add(user)
+        db.session.commit()
+    except IntegrityError:
+        print("duplicate user", user.name)
+        db.session.rollback()
 
-    user_broadcast_services = Service.query.filter(
-        Service.permissions.any(permission='broadcast'),
-        Service.users.any(id=user_id)
-    )
 
-    for service in user_broadcast_services:
-        permission_list = [
-            Permission(service_id=service.id, user_id=user_id, permission=permission)
-            for permission in [
-                'reject_broadcasts', 'cancel_broadcasts',  # required to create / approve
-                'create_broadcasts', 'approve_broadcasts',  # minimum for testing
-                'manage_templates',  # unlikely but might be useful
-                'view_activity',  # normally added on invite / service creation
-            ]
-        ]
+@notify_command(name='create-admin-jwt')
+def create_admin_jwt():
+    if getenv('NOTIFY_ENVIRONMENT', '') != 'development':
+        current_app.logger.error('Can only be run in development')
+        return
+    print(create_jwt_token(current_app.config['SECRET_KEY'], current_app.config['ADMIN_CLIENT_ID']))
 
-        permission_dao.set_user_service_permission(
-            user, service, permission_list, _commit=True, replace=True
-        )
+
+@notify_command(name='create-user-jwt')
+@click.option('-t', '--token', required=True, prompt=False)
+def create_user_jwt(token):
+    if getenv('NOTIFY_ENVIRONMENT', '') != 'development':
+        current_app.logger.error('Can only be run in development')
+        return
+    service_id = token[-73:-37]
+    api_key = token[-36:]
+    print(create_jwt_token(api_key, service_id))
+
+
+def _update_template(id, name, template_type, content, subject):
+
+    template = Template.query.filter_by(id=id).first()
+    template.name = name
+    template.template_type = template_type
+    template.content = '\n'.join(content)
+    template.subject = subject
+
+    history = TemplateHistory.query.filter_by(id=id).first()
+    history.name = name
+    history.template_type = template_type
+    history.content = '\n'.join(content)
+    history.subject = subject
+
+    db.session.commit()
+
+
+@notify_command(name='update-templates')
+def update_templates():
+    with open(current_app.config['CONFIG_FILES'] + '/templates.json') as f:
+        data = json.load(f)
+        for d in data:
+            _update_template(d['id'], d['name'], d['type'], d['content'], d['subject'])

app/complaint/complaint_schema.py

@@ -1,6 +1,6 @@
 
 complaint_count_request = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
+    "$schema": "http://json-schema.org/draft-07/schema#",
     "description": "complaint count request schema",
     "type": "object",
     "title": "Complaint count request",

app/config.py

@@ -1,16 +1,12 @@
 import json
-import os
 from datetime import timedelta
+from os import getenv, path
 
+import notifications_utils
 from celery.schedules import crontab
 from kombu import Exchange, Queue
 
-if os.environ.get('VCAP_SERVICES'):
-    # on cloudfoundry, config is a json blob in VCAP_SERVICES - unpack it, and populate
-    # standard environment variables from it
-    from app.cloudfoundry_config import extract_cloudfoundry_config
-
-    extract_cloudfoundry_config()
+from app.cloudfoundry_config import cloud_config
 
 
 class QueueNames(object):
@@ -18,24 +14,19 @@ class QueueNames(object):
     PRIORITY = 'priority-tasks'
     DATABASE = 'database-tasks'
     SEND_SMS = 'send-sms-tasks'
+    CHECK_SMS = 'check-sms_tasks'
     SEND_EMAIL = 'send-email-tasks'
     RESEARCH_MODE = 'research-mode-tasks'
     REPORTING = 'reporting-tasks'
     JOBS = 'job-tasks'
     RETRY = 'retry-tasks'
     NOTIFY = 'notify-internal-tasks'
-    PROCESS_FTP = 'process-ftp-tasks'
-    CREATE_LETTERS_PDF = 'create-letters-pdf-tasks'
     CALLBACKS = 'service-callbacks'
     CALLBACKS_RETRY = 'service-callbacks-retry'
-    LETTERS = 'letter-tasks'
     SMS_CALLBACKS = 'sms-callbacks'
     ANTIVIRUS = 'antivirus-tasks'
-    SANITISE_LETTERS = 'sanitise-letter-tasks'
     SAVE_API_EMAIL = 'save-api-email-tasks'
     SAVE_API_SMS = 'save-api-sms-tasks'
-    BROADCASTS = 'broadcast-tasks'
-    GOVUK_ALERTS = 'govuk-alerts'
 
     @staticmethod
     def all_queues():
@@ -44,158 +35,134 @@ class QueueNames(object):
             QueueNames.PERIODIC,
             QueueNames.DATABASE,
             QueueNames.SEND_SMS,
+            QueueNames.CHECK_SMS,
             QueueNames.SEND_EMAIL,
             QueueNames.RESEARCH_MODE,
             QueueNames.REPORTING,
             QueueNames.JOBS,
             QueueNames.RETRY,
             QueueNames.NOTIFY,
-            QueueNames.CREATE_LETTERS_PDF,
             QueueNames.CALLBACKS,
             QueueNames.CALLBACKS_RETRY,
-            QueueNames.LETTERS,
             QueueNames.SMS_CALLBACKS,
             QueueNames.SAVE_API_EMAIL,
             QueueNames.SAVE_API_SMS,
-            QueueNames.BROADCASTS,
         ]
 
 
-class BroadcastProvider:
-    EE = 'ee'
-    VODAFONE = 'vodafone'
-    THREE = 'three'
-    O2 = 'o2'
-
-    PROVIDERS = [EE, VODAFONE, THREE, O2]
-
-
 class TaskNames(object):
     PROCESS_INCOMPLETE_JOBS = 'process-incomplete-jobs'
-    ZIP_AND_SEND_LETTER_PDFS = 'zip-and-send-letter-pdfs'
     SCAN_FILE = 'scan-file'
-    SANITISE_LETTER = 'sanitise-and-upload-letter'
-    CREATE_PDF_FOR_TEMPLATED_LETTER = 'create-pdf-for-templated-letter'
-    PUBLISH_GOVUK_ALERTS = 'publish-govuk-alerts'
-    RECREATE_PDF_FOR_PRECOMPILED_LETTER = 'recreate-pdf-for-precompiled-letter'
 
 
 class Config(object):
-    # URL of admin app
-    ADMIN_BASE_URL = os.getenv('ADMIN_BASE_URL', 'http://localhost:6012')
-
-    # URL of api app (on AWS this is the internal api endpoint)
-    API_HOST_NAME = os.getenv('API_HOST_NAME')
-
-    # secrets that internal apps, such as the admin app or document download, must use to authenticate with the API
-    ADMIN_CLIENT_ID = 'notify-admin'
-    GOVUK_ALERTS_CLIENT_ID = 'govuk-alerts'
-
-    INTERNAL_CLIENT_API_KEYS = json.loads(
-        os.environ.get('INTERNAL_CLIENT_API_KEYS', '{}')
-    )
-
-    # encyption secret/salt
-    SECRET_KEY = os.getenv('SECRET_KEY')
-    DANGEROUS_SALT = os.getenv('DANGEROUS_SALT')
-
-    # DB conection string
-    SQLALCHEMY_DATABASE_URI = os.getenv('SQLALCHEMY_DATABASE_URI')
-
-    # MMG API Key
-    MMG_API_KEY = os.getenv('MMG_API_KEY')
-
-    # Firetext API Key
-    FIRETEXT_API_KEY = os.getenv("FIRETEXT_API_KEY")
-    FIRETEXT_INTERNATIONAL_API_KEY = os.getenv("FIRETEXT_INTERNATIONAL_API_KEY", "placeholder")
-
-    # Prefix to identify queues in SQS
-    NOTIFICATION_QUEUE_PREFIX = os.getenv('NOTIFICATION_QUEUE_PREFIX')
-
-    # URL of redis instance
-    REDIS_URL = os.getenv('REDIS_URL')
-    REDIS_ENABLED = os.getenv('REDIS_ENABLED') == '1'
-    EXPIRE_CACHE_TEN_MINUTES = 600
-    EXPIRE_CACHE_EIGHT_DAYS = 8 * 24 * 60 * 60
-
-    # Zendesk
-    ZENDESK_API_KEY = os.environ.get('ZENDESK_API_KEY')
-
-    # Logging
-    DEBUG = False
-    NOTIFY_LOG_PATH = os.getenv('NOTIFY_LOG_PATH')
-
-    # Cronitor
-    CRONITOR_ENABLED = False
-    CRONITOR_KEYS = json.loads(os.environ.get('CRONITOR_KEYS', '{}'))
-
-    # Antivirus
-    ANTIVIRUS_ENABLED = True
-
-    ###########################
-    # Default config values ###
-    ###########################
-
-    NOTIFY_ENVIRONMENT = 'development'
-    AWS_REGION = 'eu-west-1'
-    INVITATION_EXPIRATION_DAYS = 2
     NOTIFY_APP_NAME = 'api'
+    NOTIFY_ENVIRONMENT = getenv('NOTIFY_ENVIRONMENT', 'development')
+    # URL of admin app
+    ADMIN_BASE_URL = getenv('ADMIN_BASE_URL', 'http://localhost:6012')
+    # URL of api app (on AWS this is the internal api endpoint)
+    API_HOST_NAME = getenv('API_HOST_NAME', 'http://localhost:6011')
+
+    # Credentials
+    # secrets that internal apps, such as the admin app or document download, must use to authenticate with the API
+    # ADMIN_CLIENT_ID is called ADMIN_CLIENT_USER_NAME in api repo, they should match
+    ADMIN_CLIENT_ID = getenv('ADMIN_CLIENT_ID', 'notify-admin')
+    INTERNAL_CLIENT_API_KEYS = json.loads(
+        getenv(
+            'INTERNAL_CLIENT_API_KEYS',
+            ('{"%s":["%s"]}' % (ADMIN_CLIENT_ID, getenv('ADMIN_CLIENT_SECRET')))
+            )
+    )
+    ALLOW_EXPIRED_API_TOKEN = False
+    # encyption secret/salt
+    SECRET_KEY = getenv('SECRET_KEY')
+    DANGEROUS_SALT = getenv('DANGEROUS_SALT')
+    ROUTE_SECRET_KEY_1 = getenv('ROUTE_SECRET_KEY_1', 'dev-route-secret-key-1')
+    ROUTE_SECRET_KEY_2 = getenv('ROUTE_SECRET_KEY_2', 'dev-route-secret-key-2')
+
+    # DB settings
+    SQLALCHEMY_DATABASE_URI = cloud_config.database_url
     SQLALCHEMY_RECORD_QUERIES = False
     SQLALCHEMY_TRACK_MODIFICATIONS = False
-    SQLALCHEMY_POOL_SIZE = int(os.environ.get('SQLALCHEMY_POOL_SIZE', 5))
+    SQLALCHEMY_POOL_SIZE = int(getenv('SQLALCHEMY_POOL_SIZE', 5))
     SQLALCHEMY_POOL_TIMEOUT = 30
     SQLALCHEMY_POOL_RECYCLE = 300
     SQLALCHEMY_STATEMENT_TIMEOUT = 1200
     PAGE_SIZE = 50
     API_PAGE_SIZE = 250
+    REDIS_URL = cloud_config.redis_url
+    REDIS_ENABLED = getenv('REDIS_ENABLED', '0') == '1'
+    EXPIRE_CACHE_TEN_MINUTES = 600
+    EXPIRE_CACHE_EIGHT_DAYS = 8 * 24 * 60 * 60
+
+    # AWS Settings
+    AWS_US_TOLL_FREE_NUMBER = getenv("AWS_US_TOLL_FREE_NUMBER")
+    # Whether to ignore POSTs from SNS for replies to SMS we sent
+    RECEIVE_INBOUND_SMS = False
+    NOTIFY_EMAIL_DOMAIN = cloud_config.ses_email_domain
+    SES_STUB_URL = None  # TODO: set to a URL in env and remove this to use a stubbed SES service
+    # AWS SNS topics for delivery receipts
+    VALIDATE_SNS_TOPICS = True
+    VALID_SNS_TOPICS = cloud_config.sns_topic_arns
+
+    # these should always add up to 100%
+    SMS_PROVIDER_RESTING_POINTS = {
+        'sns': 100,
+    }
+
+    # Zendesk
+    ZENDESK_API_KEY = getenv('ZENDESK_API_KEY')
+
+    # Logging
+    DEBUG = False
+
+    # Monitoring
+    CRONITOR_ENABLED = False
+    CRONITOR_KEYS = json.loads(getenv('CRONITOR_KEYS', '{}'))
+
+    # Antivirus
+    ANTIVIRUS_ENABLED = getenv('ANTIVIRUS_ENABLED', '1') == '1'
+
+    SENDING_NOTIFICATIONS_TIMEOUT_PERIOD = 259200  # 3 days
+    INVITATION_EXPIRATION_DAYS = 2
     TEST_MESSAGE_FILENAME = 'Test message'
     ONE_OFF_MESSAGE_FILENAME = 'Report'
     MAX_VERIFY_CODE_COUNT = 5
     MAX_FAILED_LOGIN_COUNT = 10
+    API_RATE_LIMIT_ENABLED = True
 
-    # be careful increasing this size without being sure that we won't see slowness in pysftp
-    MAX_LETTER_PDF_ZIP_FILESIZE = 40 * 1024 * 1024  # 40mb
-    MAX_LETTER_PDF_COUNT_PER_ZIP = 500
-
-    CHECK_PROXY_HEADER = False
-
-    # these should always add up to 100%
-    SMS_PROVIDER_RESTING_POINTS = {
-        'mmg': 60,
-        'firetext': 40
-    }
+    # Default data
+    CONFIG_FILES = path.dirname(__file__) + '/config_files/'
 
     NOTIFY_SERVICE_ID = 'd6aa2c68-a2d9-4437-ab19-3ae8eb202553'
     NOTIFY_USER_ID = '6af522d0-2915-4e52-83a3-3690455a5fe6'
     INVITATION_EMAIL_TEMPLATE_ID = '4f46df42-f795-4cc4-83bb-65ca312f49cc'
-    BROADCAST_INVITATION_EMAIL_TEMPLATE_ID = '46152f7c-6901-41d5-8590-a5624d0d4359'
     SMS_CODE_TEMPLATE_ID = '36fb0730-6259-4da1-8a80-c8de22ad4246'
     EMAIL_2FA_TEMPLATE_ID = '299726d2-dba6-42b8-8209-30e1d66ea164'
     NEW_USER_EMAIL_VERIFICATION_TEMPLATE_ID = 'ece42649-22a8-4d06-b87f-d52d5d3f0a27'
-    PASSWORD_RESET_TEMPLATE_ID = '474e9242-823b-4f99-813d-ed392e7f1201'
+    PASSWORD_RESET_TEMPLATE_ID = '474e9242-823b-4f99-813d-ed392e7f1201'  # nosec B105 - this is not a password
     ALREADY_REGISTERED_EMAIL_TEMPLATE_ID = '0880fbb1-a0c6-46f0-9a8e-36c986381ceb'
     CHANGE_EMAIL_CONFIRMATION_TEMPLATE_ID = 'eb4d9930-87ab-4aef-9bce-786762687884'
     SERVICE_NOW_LIVE_TEMPLATE_ID = '618185c6-3636-49cd-b7d2-6f6f5eb3bdde'
-    ORGANISATION_INVITATION_EMAIL_TEMPLATE_ID = '203566f0-d835-47c5-aa06-932439c86573'
+    ORGANIZATION_INVITATION_EMAIL_TEMPLATE_ID = '203566f0-d835-47c5-aa06-932439c86573'
     TEAM_MEMBER_EDIT_EMAIL_TEMPLATE_ID = 'c73f1d71-4049-46d5-a647-d013bdeca3f0'
     TEAM_MEMBER_EDIT_MOBILE_TEMPLATE_ID = '8a31520f-4751-4789-8ea1-fe54496725eb'
     REPLY_TO_EMAIL_ADDRESS_VERIFICATION_TEMPLATE_ID = 'a42f1d17-9404-46d5-a647-d013bdfca3e1'
     MOU_SIGNER_RECEIPT_TEMPLATE_ID = '4fd2e43c-309b-4e50-8fb8-1955852d9d71'
     MOU_SIGNED_ON_BEHALF_SIGNER_RECEIPT_TEMPLATE_ID = 'c20206d5-bf03-4002-9a90-37d5032d9e84'
     MOU_SIGNED_ON_BEHALF_ON_BEHALF_RECEIPT_TEMPLATE_ID = '522b6657-5ca5-4368-a294-6b527703bd0b'
-    NOTIFY_INTERNATIONAL_SMS_SENDER = '07984404008'
+    NOTIFY_INTERNATIONAL_SMS_SENDER = getenv('AWS_US_TOLL_FREE_NUMBER')
     LETTERS_VOLUME_EMAIL_TEMPLATE_ID = '11fad854-fd38-4a7c-bd17-805fb13dfc12'
+    NHS_EMAIL_BRANDING_ID = 'a7dc4e56-660b-4db7-8cff-12c37b12b5ea'
     # we only need real email in Live environment (production)
-    DVLA_EMAIL_ADDRESSES = json.loads(os.environ.get('DVLA_EMAIL_ADDRESSES', '[]'))
+    DVLA_EMAIL_ADDRESSES = json.loads(getenv('DVLA_EMAIL_ADDRESSES', '[]'))
 
     CELERY = {
-        'broker_url': 'sqs://',
+        'broker_url': REDIS_URL,
         'broker_transport_options': {
-            'region': AWS_REGION,
             'visibility_timeout': 310,
-            'queue_name_prefix': NOTIFICATION_QUEUE_PREFIX,
         },
-        'timezone': 'Europe/London',
+        'timezone': getenv("TIMEZONE", 'UTC'),
         'imports': [
             'app.celery.tasks',
             'app.celery.scheduled_tasks',
@@ -223,11 +190,6 @@ class Config(object):
                 'schedule': timedelta(minutes=66),
                 'options': {'queue': QueueNames.PERIODIC}
             },
-            'switch-current-sms-provider-on-slow-delivery': {
-                'task': 'switch-current-sms-provider-on-slow-delivery',
-                'schedule': crontab(),  # Every minute
-                'options': {'queue': QueueNames.PERIODIC}
-            },
             'check-job-status': {
                 'task': 'check-job-status',
                 'schedule': crontab(),
@@ -251,200 +213,108 @@ class Config(object):
             # app/celery/nightly_tasks.py
             'timeout-sending-notifications': {
                 'task': 'timeout-sending-notifications',
-                'schedule': crontab(hour=0, minute=5),
+                'schedule': crontab(hour=4, minute=5),
                 'options': {'queue': QueueNames.PERIODIC}
             },
             'create-nightly-billing': {
                 'task': 'create-nightly-billing',
-                'schedule': crontab(hour=0, minute=15),
+                'schedule': crontab(hour=4, minute=15),
                 'options': {'queue': QueueNames.REPORTING}
             },
             'create-nightly-notification-status': {
                 'task': 'create-nightly-notification-status',
-                'schedule': crontab(hour=0, minute=30),  # after 'timeout-sending-notifications'
+                'schedule': crontab(hour=4, minute=30),  # after 'timeout-sending-notifications'
                 'options': {'queue': QueueNames.REPORTING}
             },
             'delete-notifications-older-than-retention': {
                 'task': 'delete-notifications-older-than-retention',
-                'schedule': crontab(hour=3, minute=0),  # after 'create-nightly-notification-status'
+                'schedule': crontab(hour=7, minute=0),  # after 'create-nightly-notification-status'
                 'options': {'queue': QueueNames.REPORTING}
             },
             'delete-inbound-sms': {
                 'task': 'delete-inbound-sms',
-                'schedule': crontab(hour=1, minute=40),
+                'schedule': crontab(hour=5, minute=40),
                 'options': {'queue': QueueNames.PERIODIC}
             },
             'save-daily-notification-processing-time': {
                 'task': 'save-daily-notification-processing-time',
-                'schedule': crontab(hour=2, minute=0),
+                'schedule': crontab(hour=6, minute=0),
+                'options': {'queue': QueueNames.PERIODIC}
+            },
+            'cleanup-unfinished-jobs': {
+                'task': 'cleanup-unfinished-jobs',
+                'schedule': crontab(hour=4, minute=5),
                 'options': {'queue': QueueNames.PERIODIC}
             },
             'remove_sms_email_jobs': {
                 'task': 'remove_sms_email_jobs',
-                'schedule': crontab(hour=4, minute=0),
+                'schedule': crontab(hour=8, minute=0),
                 'options': {'queue': QueueNames.PERIODIC},
             },
-            'remove_letter_jobs': {
-                'task': 'remove_letter_jobs',
-                'schedule': crontab(hour=4, minute=20),
-                # since we mark jobs as archived
-                'options': {'queue': QueueNames.PERIODIC},
-            },
-            'check-if-letters-still-in-created': {
-                'task': 'check-if-letters-still-in-created',
-                'schedule': crontab(day_of_week='mon-fri', hour=7, minute=0),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'check-if-letters-still-pending-virus-check': {
-                'task': 'check-if-letters-still-pending-virus-check',
-                'schedule': crontab(day_of_week='mon-fri', hour='9,15', minute=0),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
             'check-for-services-with-high-failure-rates-or-sending-to-tv-numbers': {
                 'task': 'check-for-services-with-high-failure-rates-or-sending-to-tv-numbers',
-                'schedule': crontab(day_of_week='mon-fri', hour=10, minute=30),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'raise-alert-if-letter-notifications-still-sending': {
-                'task': 'raise-alert-if-letter-notifications-still-sending',
-                'schedule': crontab(hour=17, minute=00),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            # The collate-letter-pdf does assume it is called in an hour that BST does not make a
-            # difference to the truncate date which translates to the filename to process
-            'collate-letter-pdfs-to-be-sent': {
-                'task': 'collate-letter-pdfs-to-be-sent',
-                'schedule': crontab(hour=17, minute=50),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'raise-alert-if-no-letter-ack-file': {
-                'task': 'raise-alert-if-no-letter-ack-file',
-                'schedule': crontab(hour=23, minute=00),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'trigger-link-tests': {
-                'task': 'trigger-link-tests',
-                'schedule': timedelta(minutes=15),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'auto-expire-broadcast-messages': {
-                'task': 'auto-expire-broadcast-messages',
-                'schedule': timedelta(minutes=5),
-                'options': {'queue': QueueNames.PERIODIC}
-            },
-            'remove-yesterdays-planned-tests-on-govuk-alerts': {
-                'task': 'remove-yesterdays-planned-tests-on-govuk-alerts',
-                'schedule': crontab(hour=00, minute=00),
+                'schedule': crontab(day_of_week='mon-fri', hour=14, minute=30),
                 'options': {'queue': QueueNames.PERIODIC}
             },
         }
     }
 
     # we can set celeryd_prefetch_multiplier to be 1 for celery apps which handle only long running tasks
-    if os.getenv('CELERYD_PREFETCH_MULTIPLIER'):
-        CELERY['worker_prefetch_multiplier'] = os.getenv('CELERYD_PREFETCH_MULTIPLIER')
+    if getenv('CELERYD_PREFETCH_MULTIPLIER'):
+        CELERY['worker_prefetch_multiplier'] = getenv('CELERYD_PREFETCH_MULTIPLIER')
 
     FROM_NUMBER = 'development'
 
-    STATSD_HOST = os.getenv('STATSD_HOST')
-    STATSD_PORT = 8125
-    STATSD_ENABLED = bool(STATSD_HOST)
-
-    SENDING_NOTIFICATIONS_TIMEOUT_PERIOD = 259200  # 3 days
-
     SIMULATED_EMAIL_ADDRESSES = (
         'simulate-delivered@notifications.service.gov.uk',
         'simulate-delivered-2@notifications.service.gov.uk',
         'simulate-delivered-3@notifications.service.gov.uk',
     )
-
-    SIMULATED_SMS_NUMBERS = ('+447700900000', '+447700900111', '+447700900222')
+    SIMULATED_SMS_NUMBERS = ('+12028675000', '+12028675111', '+12028675222')
 
     FREE_SMS_TIER_FRAGMENT_COUNT = 250000
 
-    SMS_INBOUND_WHITELIST = json.loads(os.environ.get('SMS_INBOUND_WHITELIST', '[]'))
-    FIRETEXT_INBOUND_SMS_AUTH = json.loads(os.environ.get('FIRETEXT_INBOUND_SMS_AUTH', '[]'))
-    MMG_INBOUND_SMS_AUTH = json.loads(os.environ.get('MMG_INBOUND_SMS_AUTH', '[]'))
-    MMG_INBOUND_SMS_USERNAME = json.loads(os.environ.get('MMG_INBOUND_SMS_USERNAME', '[]'))
-    ROUTE_SECRET_KEY_1 = os.environ.get('ROUTE_SECRET_KEY_1', '')
-    ROUTE_SECRET_KEY_2 = os.environ.get('ROUTE_SECRET_KEY_2', '')
+    DAILY_MESSAGE_LIMIT = notifications_utils.DAILY_MESSAGE_LIMIT
 
-    HIGH_VOLUME_SERVICE = json.loads(os.environ.get('HIGH_VOLUME_SERVICE', '[]'))
+    HIGH_VOLUME_SERVICE = json.loads(getenv('HIGH_VOLUME_SERVICE', '[]'))
 
-    TEMPLATE_PREVIEW_API_HOST = os.environ.get('TEMPLATE_PREVIEW_API_HOST', 'http://localhost:6013')
-    TEMPLATE_PREVIEW_API_KEY = os.environ.get('TEMPLATE_PREVIEW_API_KEY', 'my-secret-key')
+    TEMPLATE_PREVIEW_API_HOST = getenv('TEMPLATE_PREVIEW_API_HOST', 'http://localhost:6013')
+    TEMPLATE_PREVIEW_API_KEY = getenv('TEMPLATE_PREVIEW_API_KEY', 'my-secret-key')
 
-    DOCUMENT_DOWNLOAD_API_HOST = os.environ.get('DOCUMENT_DOWNLOAD_API_HOST', 'http://localhost:7000')
-    DOCUMENT_DOWNLOAD_API_KEY = os.environ.get('DOCUMENT_DOWNLOAD_API_KEY', 'auth-token')
-
-    # these environment vars aren't defined in the manifest so to set them on paas use `cf set-env`
-    MMG_URL = os.environ.get("MMG_URL", "https://api.mmg.co.uk/jsonv2a/api.php")
-    FIRETEXT_URL = os.environ.get("FIRETEXT_URL", "https://www.firetext.co.uk/api/sendsms/json")
-    REACH_URL = os.environ.get("REACH_URL", "TODO")
-    SES_STUB_URL = os.environ.get("SES_STUB_URL")
-
-    AWS_REGION = 'eu-west-1'
-
-    CBC_PROXY_ENABLED = True
-    CBC_PROXY_AWS_ACCESS_KEY_ID = os.environ.get('CBC_PROXY_AWS_ACCESS_KEY_ID', '')
-    CBC_PROXY_AWS_SECRET_ACCESS_KEY = os.environ.get('CBC_PROXY_AWS_SECRET_ACCESS_KEY', '')
-
-    ENABLED_CBCS = {BroadcastProvider.EE, BroadcastProvider.THREE, BroadcastProvider.O2, BroadcastProvider.VODAFONE}
-
-    # as defined in api db migration 0331_add_broadcast_org.py
-    BROADCAST_ORGANISATION_ID = '38e4bf69-93b0-445d-acee-53ea53fe02df'
+    DOCUMENT_DOWNLOAD_API_HOST = getenv('DOCUMENT_DOWNLOAD_API_HOST', 'http://localhost:7000')
+    DOCUMENT_DOWNLOAD_API_KEY = getenv('DOCUMENT_DOWNLOAD_API_KEY', 'auth-token')
 
 
-######################
-# Config overrides ###
-######################
+def _s3_credentials_from_env(bucket_prefix):
+    return {
+        'bucket': getenv(f"{bucket_prefix}_BUCKET_NAME"),
+        'access_key_id': getenv(f"{bucket_prefix}_AWS_ACCESS_KEY_ID"),
+        'secret_access_key': getenv(f"{bucket_prefix}_AWS_SECRET_ACCESS_KEY"),
+        'region': getenv(f"{bucket_prefix}_AWS_REGION")
+    }
+
 
 class Development(Config):
     DEBUG = True
+    NOTIFY_LOG_LEVEL = "DEBUG"
     SQLALCHEMY_ECHO = False
-
-    CSV_UPLOAD_BUCKET_NAME = 'development-notifications-csv-upload'
-    CONTACT_LIST_BUCKET_NAME = 'development-contact-list'
-    TEST_LETTERS_BUCKET_NAME = 'development-test-letters'
-    DVLA_RESPONSE_BUCKET_NAME = 'notify.tools-ftp'
-    LETTERS_PDF_BUCKET_NAME = 'development-letters-pdf'
-    LETTERS_SCAN_BUCKET_NAME = 'development-letters-scan'
-    INVALID_PDF_BUCKET_NAME = 'development-letters-invalid-pdf'
-    TRANSIENT_UPLOADED_LETTERS = 'development-transient-uploaded-letters'
-    LETTER_SANITISE_BUCKET_NAME = 'development-letters-sanitise'
-
-    INTERNAL_CLIENT_API_KEYS = {
-        Config.ADMIN_CLIENT_ID: ['dev-notify-secret-key'],
-        Config.GOVUK_ALERTS_CLIENT_ID: ['govuk-alerts-secret-key']
-    }
-
-    SECRET_KEY = 'dev-notify-secret-key'
-    DANGEROUS_SALT = 'dev-notify-salt'
-
-    MMG_INBOUND_SMS_AUTH = ['testkey']
-    MMG_INBOUND_SMS_USERNAME = ['username']
-
-    NOTIFY_ENVIRONMENT = 'development'
-    NOTIFY_LOG_PATH = 'application.log'
-    NOTIFY_EMAIL_DOMAIN = "notify.tools"
-
-    SQLALCHEMY_DATABASE_URI = os.getenv('SQLALCHEMY_DATABASE_URI', 'postgresql://localhost/notification_api')
-    REDIS_URL = os.getenv('REDIS_URL', 'redis://localhost:6379/0')
-
-    ANTIVIRUS_ENABLED = os.getenv('ANTIVIRUS_ENABLED') == '1'
-
-    API_HOST_NAME = os.getenv('API_HOST_NAME', 'http://localhost:6011')
-    API_RATE_LIMIT_ENABLED = True
     DVLA_EMAIL_ADDRESSES = ['success@simulator.amazonses.com']
 
-    CBC_PROXY_ENABLED = False
+    # Buckets
+    CSV_UPLOAD_BUCKET = _s3_credentials_from_env('CSV')
+
+    # credential overrides
+    DANGEROUS_SALT = 'development-notify-salt'
+    SECRET_KEY = 'dev-notify-secret-key'  # nosec B105 - this is only used in development
+    INTERNAL_CLIENT_API_KEYS = {Config.ADMIN_CLIENT_ID: ['dev-notify-secret-key']}
+    ALLOW_EXPIRED_API_TOKEN = getenv('ALLOW_EXPIRED_API_TOKEN', '0') == '1'
 
 
 class Test(Development):
-    NOTIFY_EMAIL_DOMAIN = 'test.notify.com'
     FROM_NUMBER = 'testing'
-    NOTIFY_ENVIRONMENT = 'test'
     TESTING = True
+    ANTIVIRUS_ENABLED = True
+    DVLA_EMAIL_ADDRESSES = ['success@simulator.amazonses.com', 'success+2@simulator.amazonses.com']
 
     HIGH_VOLUME_SERVICE = [
         '941b6f9a-50d7-4742-8d50-f365ca74bf27',
@@ -453,104 +323,39 @@ class Test(Development):
         '10d1b9c9-0072-4fa9-ae1c-595e333841da',
     ]
 
-    CSV_UPLOAD_BUCKET_NAME = 'test-notifications-csv-upload'
-    CONTACT_LIST_BUCKET_NAME = 'test-contact-list'
-    TEST_LETTERS_BUCKET_NAME = 'test-test-letters'
-    DVLA_RESPONSE_BUCKET_NAME = 'test.notify.com-ftp'
-    LETTERS_PDF_BUCKET_NAME = 'test-letters-pdf'
-    LETTERS_SCAN_BUCKET_NAME = 'test-letters-scan'
-    INVALID_PDF_BUCKET_NAME = 'test-letters-invalid-pdf'
-    TRANSIENT_UPLOADED_LETTERS = 'test-transient-uploaded-letters'
-    LETTER_SANITISE_BUCKET_NAME = 'test-letters-sanitise'
-
-    # this is overriden in jenkins and on cloudfoundry
-    SQLALCHEMY_DATABASE_URI = os.getenv('SQLALCHEMY_DATABASE_URI', 'postgresql://localhost/test_notification_api')
+    # this is overriden in CI
+    SQLALCHEMY_DATABASE_URI = getenv('SQLALCHEMY_DATABASE_TEST_URI')
 
     CELERY = {
         **Config.CELERY,
         'broker_url': 'you-forgot-to-mock-celery-in-your-tests://'
     }
 
-    ANTIVIRUS_ENABLED = True
-
-    API_RATE_LIMIT_ENABLED = True
-    API_HOST_NAME = "http://localhost:6011"
-
-    SMS_INBOUND_WHITELIST = ['203.0.113.195']
-    FIRETEXT_INBOUND_SMS_AUTH = ['testkey']
     TEMPLATE_PREVIEW_API_HOST = 'http://localhost:9999'
 
-    MMG_URL = 'https://example.com/mmg'
-    FIRETEXT_URL = 'https://example.com/firetext'
-    REACH_URL = 'https://example.com/reach'
 
-    CBC_PROXY_ENABLED = True
-    DVLA_EMAIL_ADDRESSES = ['success@simulator.amazonses.com', 'success+2@simulator.amazonses.com']
-
-
-class Preview(Config):
-    NOTIFY_EMAIL_DOMAIN = 'notify.works'
-    NOTIFY_ENVIRONMENT = 'preview'
-    CSV_UPLOAD_BUCKET_NAME = 'preview-notifications-csv-upload'
-    CONTACT_LIST_BUCKET_NAME = 'preview-contact-list'
-    TEST_LETTERS_BUCKET_NAME = 'preview-test-letters'
-    DVLA_RESPONSE_BUCKET_NAME = 'notify.works-ftp'
-    LETTERS_PDF_BUCKET_NAME = 'preview-letters-pdf'
-    LETTERS_SCAN_BUCKET_NAME = 'preview-letters-scan'
-    INVALID_PDF_BUCKET_NAME = 'preview-letters-invalid-pdf'
-    TRANSIENT_UPLOADED_LETTERS = 'preview-transient-uploaded-letters'
-    LETTER_SANITISE_BUCKET_NAME = 'preview-letters-sanitise'
-    FROM_NUMBER = 'preview'
-    API_RATE_LIMIT_ENABLED = True
-    CHECK_PROXY_HEADER = False
-
-
-class Staging(Config):
-    NOTIFY_EMAIL_DOMAIN = 'staging-notify.works'
-    NOTIFY_ENVIRONMENT = 'staging'
-    CSV_UPLOAD_BUCKET_NAME = 'staging-notifications-csv-upload'
-    CONTACT_LIST_BUCKET_NAME = 'staging-contact-list'
-    TEST_LETTERS_BUCKET_NAME = 'staging-test-letters'
-    DVLA_RESPONSE_BUCKET_NAME = 'staging-notify.works-ftp'
-    LETTERS_PDF_BUCKET_NAME = 'staging-letters-pdf'
-    LETTERS_SCAN_BUCKET_NAME = 'staging-letters-scan'
-    INVALID_PDF_BUCKET_NAME = 'staging-letters-invalid-pdf'
-    TRANSIENT_UPLOADED_LETTERS = 'staging-transient-uploaded-letters'
-    LETTER_SANITISE_BUCKET_NAME = 'staging-letters-sanitise'
-    FROM_NUMBER = 'stage'
-    API_RATE_LIMIT_ENABLED = True
-    CHECK_PROXY_HEADER = True
-
-
-class Live(Config):
-    NOTIFY_EMAIL_DOMAIN = 'notifications.service.gov.uk'
-    NOTIFY_ENVIRONMENT = 'live'
-    CSV_UPLOAD_BUCKET_NAME = 'live-notifications-csv-upload'
-    CONTACT_LIST_BUCKET_NAME = 'production-contact-list'
-    TEST_LETTERS_BUCKET_NAME = 'production-test-letters'
-    DVLA_RESPONSE_BUCKET_NAME = 'notifications.service.gov.uk-ftp'
-    LETTERS_PDF_BUCKET_NAME = 'production-letters-pdf'
-    LETTERS_SCAN_BUCKET_NAME = 'production-letters-scan'
-    INVALID_PDF_BUCKET_NAME = 'production-letters-invalid-pdf'
-    TRANSIENT_UPLOADED_LETTERS = 'production-transient-uploaded-letters'
-    LETTER_SANITISE_BUCKET_NAME = 'production-letters-sanitise'
-    FROM_NUMBER = 'GOVUK'
-    API_RATE_LIMIT_ENABLED = True
-    CHECK_PROXY_HEADER = True
-    SES_STUB_URL = None
+class Production(Config):
+    # buckets
+    CSV_UPLOAD_BUCKET = cloud_config.s3_credentials(
+        f"notify-api-csv-upload-bucket-{Config.NOTIFY_ENVIRONMENT}")
 
+    FROM_NUMBER = 'US Notify'
     CRONITOR_ENABLED = True
 
 
-class CloudFoundryConfig(Config):
+class Staging(Production):
+    pass
+
+
+class Demo(Production):
     pass
 
 
 configs = {
     'development': Development,
     'test': Test,
-    'live': Live,
-    'production': Live,
     'staging': Staging,
-    'preview': Preview,
+    'demo': Demo,
+    'sandbox': Staging,
+    'production': Production
 }

app/config_files/templates.json

@@ -0,0 +1,274 @@
+[
+    {
+        "id": "42a23d19-504e-49bb-a95e-4976baff4757",
+        "name": "Example text message template",
+        "type": "sms",
+        "subject":  "",
+        "content": ["Hi, I’m trying out U.S. Notify. Today is ((day of week)) and my favorite color is ((color))."]
+    },
+    {
+        "id": "4f46df42-f795-4cc4-83bb-65ca312f49cc",
+        "name": "Notify invitation email",
+        "type": "email",
+        "subject":  "((user_name)) has invited you to collaborate on ((service_name)) on U.S. Notify",
+        "content": ["((user_name)) has invited you to collaborate on ((service_name)) on U.S. Notify.",
+            "",
+            "",
+            "U.S. Notify makes it easy to keep people updated by helping you send text messages and emails.",
+            "",
+            "",
+            "Click this link to create an account on U.S. Notify:",
+            "",
+            "((url))",
+            "",
+            "",
+            "This invitation will stop working at midnight tomorrow. This is to keep ((service_name)) secure."]
+    },
+    {
+        "id": "36fb0730-6259-4da1-8a80-c8de22ad4246",
+        "name": "Notify SMS verify code",
+        "type": "sms",
+        "subject": "",
+        "content": ["((verify_code)) is your U.S. Notify authentication code"]
+    },
+    {
+        "id": "474e9242-823b-4f99-813d-ed392e7f1201",
+        "name": "Notify password reset email",
+        "type": "email",
+        "subject": "Reset your U.S. Notify password",
+        "content": ["Hi ((user_name)),",
+            "",
+            "",
+            "We received a request to reset your password on U.S. Notify.",
+            "",
+            "",
+            "If you didn’t request this email, you can ignore it – your password has not been changed.",
+            "",
+            "",
+            "To reset your password, click this link:","","","((url))"]
+    },
+    {
+        "id": "299726d2-dba6-42b8-8209-30e1d66ea164",
+        "name": "Notify email verify code",
+        "type": "email",
+        "subject": "Sign in to U.S. Notify",
+        "content": ["Hi ((name)),",
+        "",
+        "",
+        "To sign in to U.S. Notify please open this link:",
+        "",
+        "",
+        "((url))"]
+    },
+    {
+        "id": "ece42649-22a8-4d06-b87f-d52d5d3f0a27",
+        "name": "Notify email verification code",
+        "type": "email",
+        "subject": "Confirm U.S. Notify registration",
+        "content": ["Hi ((name)),",
+        "",
+        "",
+        "To complete your registration for U.S. Notify please click the link below",
+        "",
+        "",
+        "((url))"]
+    },
+    {
+        "id": "0880fbb1-a0c6-46f0-9a8e-36c986381ceb",
+        "name": "Your U.S. Notify account",
+        "type": "email",
+        "subject": "Your U.S. Notify account",
+        "content": ["You already have a U.S. Notify account with this email address.",
+        "",
+        "",
+        "Sign in here: ((signin_url))",
+        "",
+        "",
+        "If you’ve forgotten your password, you can reset it here: ((forgot_password_url))",
+        "",
+        "",
+        "",
+        "",
+        "If you didn’t try to register for a U.S. Notify account recently, please let us know here: ((feedback_url))"]
+    },
+    {
+        "id": "eb4d9930-87ab-4aef-9bce-786762687884",
+        "name": "Confirm new email address",
+        "type": "email",
+        "subject": "Confirm your email address for U.S. Notify",
+        "content": ["Hi ((name)),","","","Click this link to confirm your new email address:",
+            "",
+            "",
+            "((url))",
+            "",
+            "",
+            "If you didn’t try to change the email address for your U.S. Notify account, let us know here:",
+            "",
+            "",
+            "((feedback_url))"]
+    },
+    {
+        "id": "618185c6-3636-49cd-b7d2-6f6f5eb3bdde",
+        "name": "Automated \"You’re now live\" message",
+        "type": "email",
+        "subject": "((service name)) is now live on U.S. Notify",
+        "content": ["Hi ((name)),",
+        "",
+        "",
+        "((service name)) is now live on U.S. Notify.",
+        "",
+        "",
+        "You can send up to ((message limit)) messages per day.",
+        "",
+        "",
+        "As a live service, you’ll need to know who to contact if you have a question, or something goes wrong.",
+        "",
+        "",
+        "If our system status page shows a problem, then we’ve been alerted and are working on it – you don’t need to contact us.",
+        "",
+        "",
+        "#Problems or questions out of hours",
+        "",
+        "",
+        "We offer out of hours support for emergencies.",
+        "",
+        "",
+        "It’s only an emergency if:",
+        "",
+        "*   no one in your team can log in",
+        "",
+        "*   a ‘technical difficulties’ error appears when you try to upload a file",
+        "",
+        "*   a 500 response code appears when you try to send messages using the API",
+        "",
+        "",
+        "If you have one of these emergencies, email details to:",
+        "",
+        "notify-support@gsa.gov",
+        "",
+        "",
+        "^Only use this email address for out of hours emergencies. Don’t share this address with people outside of your team.",
+        "",
+        "",
+        "We’ll get back to you within 30 minutes and give you hourly updates until the problem’s fixed.",
+        "",
+        "",
+        "For non-emergency problems or questions, use our support page and we’ll reply in office hours.",
+        "",
+        "",
+        "Thanks",
+        "",
+        "U.S. Notify team"]
+    },
+    {
+        "id": "203566f0-d835-47c5-aa06-932439c86573",
+        "name": "Notify organization invitation email",
+        "type": "email",
+        "subject": "((user_name)) has invited you to collaborate on ((organization_name)) on U.S. Notify",
+        "content": ["((user_name)) has invited you to collaborate on ((organization_name)) on U.S. Notify.","","","U.S. Notify makes it easy to keep people updated by helping you send text messages and emails.","","","Open this link to create an account on U.S. Notify:","","((url))","","","This invitation will stop working at midnight tomorrow. This is to keep ((organization_name)) secure."]
+    },
+    {
+        "id": "c73f1d71-4049-46d5-a647-d013bdeca3f0",
+        "name": "Email address changed by service manager",
+        "type": "email",
+        "subject": "Your U.S. Notify email address has changed",
+        "content":  ["Dear ((name)),","","","((servicemanagername)) changed your Notify account email address to:","","","((email address))","","","You’ll need to use this email address next time you sign in.","","","Thanks","","","U.S. Notify team"]
+    },
+    {
+        "id": "8a31520f-4751-4789-8ea1-fe54496725eb",
+        "name": "Phone number changed by service manager",
+        "type": "sms",
+        "subject": "",
+        "content": ["Your mobile number was changed by ((servicemanagername)). Next time you sign in, your U.S. Notify authentication code will be sent to this phone."]
+    },
+    {
+        "id": "a42f1d17-9404-46d5-a647-d013bdfca3e1",
+        "name": "Verify email reply-to address for a service",
+        "type": "email",
+        "subject": "Your U.S. Notify reply-to email address",
+        "content": ["Hi,","","","This address has been provided as a reply-to email address for a U.S. Notify account.","","Any replies from users to emails they receive through U.S. Notify will come back to this email address.","","","This is just a quick check to make sure the address is valid.","","","No need to reply.","","","Thanks","","","U.S. Notify team"]
+    },
+    {
+        "id": "4fd2e43c-309b-4e50-8fb8-1955852d9d71",
+        "name": "MOU Signed By Receipt",
+        "type": "email",
+        "subject": "You’ve accepted the U.S. Notify data sharing and financial agreement",
+        "content": [
+            "Hi ((signed_by_name)),",
+            "",
+            "((org_name)) has accepted the U.S. Notify data sharing and financial agreement. ",
+            "",
+            "If you need another copy of the agreement you can download it here: ((mou_link))",
+            "",
+            "",
+            "Thanks,",
+            "U.S. Notify team"
+        ]
+    },
+    {
+        "id": "c20206d5-bf03-4002-9a90-37d5032d9e84",
+        "name": "MOU Signed On Behalf Of Receipt - Signed by",
+        "type": "email",
+        "subject": "You’ve accepted the U.S. Notify data sharing and financial agreement",
+        "content": [
+            "Hi ((signed_by_name)),",
+            "",
+            "((org_name)) has accepted the U.S. Notify data sharing and financial agreement. We’ve emailed ((on_behalf_of_name)) to let them know too.",
+            "",
+            "If you need another copy of the agreement you can download it here: ((mou_link))",
+            "",
+            "",
+            "Thanks,",
+            "U.S. Notify team"
+        ]
+    },
+    {
+        "id": "522b6657-5ca5-4368-a294-6b527703bd0b",
+        "name": "MOU Signed On Behalf Of Receipt - On Behalf Of",
+        "type": "email",
+        "subject": "((org_name)) has accepted the U.S. Notify data sharing and financial agreement",
+        "content": [
+            "Hi ((on_behalf_of_name)),",
+            "",
+            "((signed_by_name)) has accepted the U.S. Notify data sharing and financial agreement on your behalf, for ((org_name)).",
+            "",
+            "U.S. Notify lets teams in the public sector send text messages and emails. It’s built and run by a team in the TTS Public Benefits Studio (part of GSA).",
+            "",
+            "If you need another copy of the agreement you can download it here: ((mou_link))",
+            "",
+            "",
+            "Thanks,",
+            "U.S. Notify team"
+        ]
+    },
+    {
+        "id": "d0e66c4c-0c50-43f0-94f5-f85b613202d4",
+        "name": "MOU Signed Notify Team Alert",
+        "type": "email",
+        "subject": "Someone signed an MOU for an org on Notify",
+        "content": [
+            "What’s up Notifiers,",
+            "",
+            "((signed_by_name)) just accepted the data sharing and financial agreement for ((org_name)).",
+            "",
+            "See how ((org_name)) is using Notify here: ((org_dashboard_link))"
+        ]
+    },
+    {
+        "id": "11fad854-fd38-4a7c-bd17-805fb13dfc12",
+        "name": "Notify daily letter volumes",
+        "type": "email",
+        "subject": "Notify letter volume for ((date)): ((total_volume)) letters, ((total_sheets)) sheets",
+        "content": [
+            "((total_volume)) letters (((total_sheets)) sheets) sent via Notify are coming in today’s batch. These include: ",
+            "",
+            "((first_class_volume)) first class letters (((first_class_sheets)) sheets).",
+            "((second_class_volume)) second class letters (((second_class_sheets)) sheets).",
+            "((international_volume)) international letters (((international_sheets)) sheets).",
+            "",
+            "Thanks",
+            "",
+            "U.S. Notify team"
+        ]
+    }
+]

app/dao/annual_billing_dao.py

@@ -2,7 +2,7 @@ from flask import current_app
 
 from app import db
 from app.dao.dao_utils import autocommit
-from app.dao.date_util import get_current_financial_year_start_year
+from app.dao.date_util import get_current_calendar_year_start_year
 from app.models import AnnualBilling
 
 
@@ -38,7 +38,7 @@ def dao_update_annual_billing_for_future_years(service_id, free_sms_fragment_lim
 def dao_get_free_sms_fragment_limit_for_year(service_id, financial_year_start=None):
 
     if not financial_year_start:
-        financial_year_start = get_current_financial_year_start_year()
+        financial_year_start = get_current_calendar_year_start_year()
 
     return AnnualBilling.query.filter_by(
         service_id=service_id,
@@ -55,58 +55,33 @@ def dao_get_all_free_sms_fragment_limit(service_id):
 
 def set_default_free_allowance_for_service(service, year_start=None):
     default_free_sms_fragment_limits = {
-        'central': {
+        'federal': {
             2020: 250_000,
             2021: 150_000,
             2022: 40_000,
         },
-        'local': {
-            2020: 25_000,
-            2021: 25_000,
-            2022: 20_000,
-        },
-        'nhs_central': {
+        'state': {
             2020: 250_000,
             2021: 150_000,
             2022: 40_000,
         },
-        'nhs_local': {
-            2020: 25_000,
-            2021: 25_000,
-            2022: 20_000,
-        },
-        'nhs_gp': {
-            2020: 25_000,
-            2021: 10_000,
-            2022: 10_000,
-        },
-        'emergency_service': {
-            2020: 25_000,
-            2021: 25_000,
-            2022: 20_000,
-        },
-        'school_or_college': {
-            2020: 25_000,
-            2021: 10_000,
-            2022: 10_000,
-        },
         'other': {
-            2020: 25_000,
-            2021: 10_000,
-            2022: 10_000,
-        },
+            2020: 250_000,
+            2021: 150_000,
+            2022: 40_000,
+        }
     }
     if not year_start:
-        year_start = get_current_financial_year_start_year()
+        year_start = get_current_calendar_year_start_year()
     # handle cases where the year is less than 2020 or greater than 2021
     if year_start < 2020:
         year_start = 2020
     if year_start > 2022:
         year_start = 2022
-    if service.organisation_type:
-        free_allowance = default_free_sms_fragment_limits[service.organisation_type][year_start]
+    if service.organization_type:
+        free_allowance = default_free_sms_fragment_limits[service.organization_type][year_start]
     else:
-        current_app.logger.info(f"no organisation type for service {service.id}. Using other default of "
+        current_app.logger.info(f"no organization type for service {service.id}. Using other default of "
                                 f"{default_free_sms_fragment_limits['other'][year_start]}")
         free_allowance = default_free_sms_fragment_limits['other'][year_start]
 

app/dao/broadcast_message_dao.py

@@ -1,105 +0,0 @@
-import uuid
-from datetime import datetime
-
-from sqlalchemy import desc
-
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import (
-    BroadcastEvent,
-    BroadcastMessage,
-    BroadcastProvider,
-    BroadcastProviderMessage,
-    BroadcastProviderMessageNumber,
-    BroadcastProviderMessageStatus,
-    BroadcastStatusType,
-    ServiceBroadcastSettings,
-)
-
-
-def dao_get_broadcast_message_by_id_and_service_id(broadcast_message_id, service_id):
-    return BroadcastMessage.query.filter(
-        BroadcastMessage.id == broadcast_message_id,
-        BroadcastMessage.service_id == service_id
-    ).one()
-
-
-def dao_get_broadcast_message_by_references_and_service_id(references_to_original_broadcast, service_id):
-    return BroadcastMessage.query.filter(
-        BroadcastMessage.status.in_((
-            BroadcastStatusType.PENDING_APPROVAL,
-            BroadcastStatusType.BROADCASTING,
-        )),
-        BroadcastMessage.reference.in_(references_to_original_broadcast),
-        BroadcastMessage.service_id == service_id
-    ).one()
-
-
-def dao_get_broadcast_event_by_id(broadcast_event_id):
-    return BroadcastEvent.query.filter(BroadcastEvent.id == broadcast_event_id).one()
-
-
-def dao_get_broadcast_messages_for_service(service_id):
-    return BroadcastMessage.query.filter(
-        BroadcastMessage.service_id == service_id
-    ).order_by(BroadcastMessage.created_at)
-
-
-def dao_get_all_broadcast_messages():
-    return db.session.query(
-        BroadcastMessage.id,
-        BroadcastMessage.reference,
-        ServiceBroadcastSettings.channel,
-        BroadcastMessage.content,
-        BroadcastMessage.areas,
-        BroadcastMessage.status,
-        BroadcastMessage.starts_at,
-        BroadcastMessage.finishes_at,
-        BroadcastMessage.approved_at,
-        BroadcastMessage.cancelled_at,
-    ).join(
-        ServiceBroadcastSettings, ServiceBroadcastSettings.service_id == BroadcastMessage.service_id
-    ).filter(
-        BroadcastMessage.starts_at >= datetime(2021, 5, 25, 0, 0, 0),
-        BroadcastMessage.stubbed == False,  # noqa
-        BroadcastMessage.status.in_(BroadcastStatusType.LIVE_STATUSES)
-    ).order_by(desc(BroadcastMessage.starts_at)).all()
-
-
-def get_earlier_events_for_broadcast_event(broadcast_event_id):
-    """
-    This is used to build up the references list.
-    """
-    this_event = BroadcastEvent.query.get(broadcast_event_id)
-
-    return BroadcastEvent.query.filter(
-        BroadcastEvent.broadcast_message_id == this_event.broadcast_message_id,
-        BroadcastEvent.sent_at < this_event.sent_at
-    ).order_by(
-        BroadcastEvent.sent_at.asc()
-    ).all()
-
-
-@autocommit
-def create_broadcast_provider_message(broadcast_event, provider):
-    broadcast_provider_message_id = uuid.uuid4()
-    provider_message = BroadcastProviderMessage(
-        id=broadcast_provider_message_id,
-        broadcast_event=broadcast_event,
-        provider=provider,
-        status=BroadcastProviderMessageStatus.SENDING,
-    )
-    db.session.add(provider_message)
-    db.session.commit()
-    provider_message_number = None
-    if provider == BroadcastProvider.VODAFONE:
-        provider_message_number = BroadcastProviderMessageNumber(
-            broadcast_provider_message_id=broadcast_provider_message_id)
-        db.session.add(provider_message_number)
-        db.session.commit()
-    return provider_message
-
-
-@autocommit
-def update_broadcast_provider_message_status(broadcast_provider_message, *, status):
-    broadcast_provider_message.status = status

app/dao/broadcast_service_dao.py

@@ -1,100 +0,0 @@
-from datetime import datetime
-
-from flask import current_app
-
-from app import db
-from app.dao.dao_utils import autocommit, version_class
-from app.models import (
-    BROADCAST_TYPE,
-    EMAIL_AUTH_TYPE,
-    INVITE_PENDING,
-    VIEW_ACTIVITY,
-    ApiKey,
-    InvitedUser,
-    Organisation,
-    Permission,
-    Service,
-    ServiceBroadcastSettings,
-    ServicePermission,
-)
-
-
-@autocommit
-@version_class(Service)
-def set_broadcast_service_type(service, service_mode, broadcast_channel, provider_restriction):
-    insert_or_update_service_broadcast_settings(
-        service, channel=broadcast_channel, provider_restriction=provider_restriction
-    )
-
-    # Remove all permissions and add broadcast permission
-    if not service.has_permission(BROADCAST_TYPE):
-        service_permission = ServicePermission(service_id=service.id, permission=BROADCAST_TYPE)
-        db.session.add(service_permission)
-
-    ServicePermission.query.filter(
-        ServicePermission.service_id == service.id,
-        ServicePermission.permission != BROADCAST_TYPE,
-        # Email auth is an exception to the other service permissions (which relate to what type
-        # of notifications a service can send) where a broadcast service is allowed to have the
-        # email auth permission (but doesn't have to)
-        ServicePermission.permission != EMAIL_AUTH_TYPE
-    ).delete()
-
-    # Refresh the service object as it has references to the service permissions but we don't yet
-    # want to commit the permission changes incase all of this needs to rollback
-    db.session.refresh(service)
-
-    # Set service count as live false always
-    service.count_as_live = False
-
-    # Set service into training mode or live mode
-    if service_mode == "live":
-        if service.restricted:
-            # Only update the go live at timestamp if this if moving from training mode
-            # to live mode, not if it's moving from one type of live mode service to another
-            service.go_live_at = datetime.utcnow()
-        service.restricted = False
-    else:
-        service.restricted = True
-        service.go_live_at = None
-
-    # Remove all user permissions apart from view_activity for the service users and invited users
-    Permission.query.filter(
-        Permission.service_id == service.id,
-        Permission.permission != VIEW_ACTIVITY
-    ).delete()
-    InvitedUser.query.filter_by(
-        service_id=service.id,
-        status=INVITE_PENDING
-    ).update({'permissions': VIEW_ACTIVITY})
-
-    # Revoke any API keys to avoid a regular API key being used to send alerts
-    ApiKey.query.filter_by(
-        service_id=service.id,
-        expiry_date=None,
-    ).update({
-        ApiKey.expiry_date: datetime.utcnow()
-    })
-
-    # Add service to organisation
-    organisation = Organisation.query.filter_by(
-        id=current_app.config['BROADCAST_ORGANISATION_ID']
-    ).one()
-    service.organisation_id = organisation.id
-    service.organisation_type = organisation.organisation_type
-    service.crown = organisation.crown
-
-    db.session.add(service)
-
-
-def insert_or_update_service_broadcast_settings(service, channel, provider_restriction="all"):
-    if not service.service_broadcast_settings:
-        settings = ServiceBroadcastSettings()
-        settings.service = service
-        settings.channel = channel
-        settings.provider = provider_restriction
-        db.session.add(settings)
-    else:
-        service.service_broadcast_settings.channel = channel
-        service.service_broadcast_settings.provider = provider_restriction
-        db.session.add(service.service_broadcast_settings)

app/dao/complaint_dao.py

@@ -6,7 +6,7 @@ from sqlalchemy import desc
 from app import db
 from app.dao.dao_utils import autocommit
 from app.models import Complaint
-from app.utils import get_london_midnight_in_utc
+from app.utils import get_midnight_in_utc
 
 
 @autocommit
@@ -28,7 +28,7 @@ def fetch_complaints_by_service(service_id):
 
 
 def fetch_count_of_complaints(start_date, end_date):
-    start_date = get_london_midnight_in_utc(start_date)
-    end_date = get_london_midnight_in_utc(end_date + timedelta(days=1))
+    start_date = get_midnight_in_utc(start_date)
+    end_date = get_midnight_in_utc(end_date + timedelta(days=1))
 
     return Complaint.query.filter(Complaint.created_at >= start_date, Complaint.created_at < end_date).count()

app/dao/daily_sorted_letter_dao.py

@@ -1,38 +0,0 @@
-from datetime import datetime
-
-from sqlalchemy.dialects.postgresql import insert
-
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import DailySortedLetter
-
-
-def dao_get_daily_sorted_letter_by_billing_day(billing_day):
-    return DailySortedLetter.query.filter_by(
-        billing_day=billing_day
-    ).first()
-
-
-@autocommit
-def dao_create_or_update_daily_sorted_letter(new_daily_sorted_letter):
-    '''
-    This uses the Postgres upsert to avoid race conditions when two threads try and insert
-    at the same row. The excluded object refers to values that we tried to insert but were
-    rejected.
-    http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#insert-on-conflict-upsert
-    '''
-    table = DailySortedLetter.__table__
-    stmt = insert(table).values(
-        billing_day=new_daily_sorted_letter.billing_day,
-        file_name=new_daily_sorted_letter.file_name,
-        unsorted_count=new_daily_sorted_letter.unsorted_count,
-        sorted_count=new_daily_sorted_letter.sorted_count)
-    stmt = stmt.on_conflict_do_update(
-        index_elements=[table.c.billing_day, table.c.file_name],
-        set_={
-            'unsorted_count': stmt.excluded.unsorted_count,
-            'sorted_count': stmt.excluded.sorted_count,
-            'updated_at': datetime.utcnow()
-        }
-    )
-    db.session.connection().execute(stmt)

app/dao/date_util.py

@@ -1,16 +1,13 @@
 from datetime import date, datetime, time, timedelta
 
-import pytz
-from notifications_utils.timezones import convert_bst_to_utc
-
 
 def get_months_for_financial_year(year):
     return [
-        convert_bst_to_utc(month) for month in (
+        month for month in (
             get_months_for_year(4, 13, year)
             + get_months_for_year(1, 4, year + 1)
         )
-        if convert_bst_to_utc(month) < datetime.now()
+        if month < datetime.now()
     ]
 
 
@@ -18,27 +15,28 @@ def get_months_for_year(start, end, year):
     return [datetime(year, month, 1) for month in range(start, end)]
 
 
-def get_financial_year(year):
-    return get_april_fools(year), get_april_fools(year + 1) - timedelta(microseconds=1)
+def get_calendar_year(year):
+    return get_new_years(year), get_new_years(year + 1) - timedelta(microseconds=1)
 
 
-def get_current_financial_year():
+def get_calendar_year_dates(year):
+    year_start_datetime, year_end_datetime = get_calendar_year(year)
+
+    return (
+        year_start_datetime.date(),
+        year_end_datetime.date()
+    )
+
+
+def get_current_calendar_year():
     now = datetime.utcnow()
-    current_month = int(now.strftime('%-m'))
     current_year = int(now.strftime('%Y'))
-    year = current_year if current_month > 3 else current_year - 1
-    return get_financial_year(year)
+    year = current_year
+    return get_calendar_year(year)
 
 
-def get_april_fools(year):
-    """
-     This function converts the start of the financial year April 1, 00:00 as BST (British Standard Time) to UTC,
-     the tzinfo is lastly removed from the datetime because the database stores the timestamps without timezone.
-     :param year: the year to calculate the April 1, 00:00 BST for
-     :return: the datetime of April 1 for the given year, for example 2016 = 2016-03-31 23:00:00
-    """
-    return pytz.timezone('Europe/London').localize(datetime(year, 4, 1, 0, 0, 0)).astimezone(pytz.UTC).replace(
-        tzinfo=None)
+def get_new_years(year):
+    return datetime(year, 1, 1, 0, 0, 0)
 
 
 def get_month_start_and_end_date_in_utc(month_year):
@@ -51,24 +49,24 @@ def get_month_start_and_end_date_in_utc(month_year):
     _, num_days = calendar.monthrange(month_year.year, month_year.month)
     first_day = datetime(month_year.year, month_year.month, 1, 0, 0, 0)
     last_day = datetime(month_year.year, month_year.month, num_days, 23, 59, 59, 99999)
-    return convert_bst_to_utc(first_day), convert_bst_to_utc(last_day)
+    return first_day, last_day
 
 
-def get_current_financial_year_start_year():
+def get_current_calendar_year_start_year():
     now = datetime.now()
     financial_year_start = now.year
-    start_date, end_date = get_financial_year(now.year)
+    start_date, end_date = get_calendar_year(now.year)
     if now < start_date:
         financial_year_start = financial_year_start - 1
     return financial_year_start
 
 
-def get_financial_year_for_datetime(start_date):
+def get_calendar_year_for_datetime(start_date):
     if type(start_date) == date:
         start_date = datetime.combine(start_date, time.min)
 
     year = int(start_date.strftime('%Y'))
-    if start_date < get_april_fools(year):
+    if start_date < get_new_years(year):
         return year - 1
     else:
         return year

app/dao/fact_notification_status_dao.py

@@ -23,65 +23,56 @@ from app.models import (
     NOTIFICATION_TEMPORARY_FAILURE,
     FactNotificationStatus,
     Notification,
+    NotificationAllTimeView,
     Service,
     Template,
 )
 from app.utils import (
-    get_london_midnight_in_utc,
-    get_london_month_from_utc_column,
-    get_notification_table_to_use,
+    get_midnight_in_utc,
+    get_month_from_utc_column,
     midnight_n_days_ago,
 )
 
 
 @autocommit
 def update_fact_notification_status(process_day, notification_type, service_id):
-    start_date = get_london_midnight_in_utc(process_day)
-    end_date = get_london_midnight_in_utc(process_day + timedelta(days=1))
+    start_date = get_midnight_in_utc(process_day)
+    end_date = get_midnight_in_utc(process_day + timedelta(days=1))
 
     # delete any existing rows in case some no longer exist e.g. if all messages are sent
     FactNotificationStatus.query.filter(
-        FactNotificationStatus.bst_date == process_day,
+        FactNotificationStatus.local_date == process_day,
         FactNotificationStatus.notification_type == notification_type,
         FactNotificationStatus.service_id == service_id,
     ).delete()
 
-    # query notifications or notification_history for the day, depending on their data retention
-    service = Service.query.get(service_id)
-    source_table = get_notification_table_to_use(
-        service,
-        notification_type,
-        process_day,
-        has_delete_task_run=False
-    )
-
     query = db.session.query(
         literal(process_day).label("process_day"),
-        source_table.template_id,
+        NotificationAllTimeView.template_id,
         literal(service_id).label("service_id"),
-        func.coalesce(source_table.job_id, '00000000-0000-0000-0000-000000000000').label('job_id'),
+        func.coalesce(NotificationAllTimeView.job_id, '00000000-0000-0000-0000-000000000000').label('job_id'),
         literal(notification_type).label("notification_type"),
-        source_table.key_type,
-        source_table.status,
+        NotificationAllTimeView.key_type,
+        NotificationAllTimeView.status,
         func.count().label('notification_count')
     ).filter(
-        source_table.created_at >= start_date,
-        source_table.created_at < end_date,
-        source_table.notification_type == notification_type,
-        source_table.service_id == service_id,
-        source_table.key_type.in_((KEY_TYPE_NORMAL, KEY_TYPE_TEAM)),
+        NotificationAllTimeView.created_at >= start_date,
+        NotificationAllTimeView.created_at < end_date,
+        NotificationAllTimeView.notification_type == notification_type,
+        NotificationAllTimeView.service_id == service_id,
+        NotificationAllTimeView.key_type.in_((KEY_TYPE_NORMAL, KEY_TYPE_TEAM)),
     ).group_by(
-        source_table.template_id,
-        source_table.template_id,
+        NotificationAllTimeView.template_id,
+        NotificationAllTimeView.template_id,
         'job_id',
-        source_table.key_type,
-        source_table.status
+        NotificationAllTimeView.key_type,
+        NotificationAllTimeView.status
     )
 
     db.session.connection().execute(
         insert(FactNotificationStatus.__table__).from_select(
             [
-                FactNotificationStatus.bst_date,
+                FactNotificationStatus.local_date,
                 FactNotificationStatus.template_id,
                 FactNotificationStatus.service_id,
                 FactNotificationStatus.job_id,
@@ -97,32 +88,32 @@ def update_fact_notification_status(process_day, notification_type, service_id):
 
 def fetch_notification_status_for_service_by_month(start_date, end_date, service_id):
     return db.session.query(
-        func.date_trunc('month', FactNotificationStatus.bst_date).label('month'),
+        func.date_trunc('month', FactNotificationStatus.local_date).label('month'),
         FactNotificationStatus.notification_type,
         FactNotificationStatus.notification_status,
         func.sum(FactNotificationStatus.notification_count).label('count')
     ).filter(
         FactNotificationStatus.service_id == service_id,
-        FactNotificationStatus.bst_date >= start_date,
-        FactNotificationStatus.bst_date < end_date,
+        FactNotificationStatus.local_date >= start_date,
+        FactNotificationStatus.local_date < end_date,
         FactNotificationStatus.key_type != KEY_TYPE_TEST
     ).group_by(
-        func.date_trunc('month', FactNotificationStatus.bst_date).label('month'),
+        func.date_trunc('month', FactNotificationStatus.local_date).label('month'),
         FactNotificationStatus.notification_type,
         FactNotificationStatus.notification_status
     ).all()
 
 
-def fetch_notification_status_for_service_for_day(bst_day, service_id):
+def fetch_notification_status_for_service_for_day(fetch_day, service_id):
     return db.session.query(
         # return current month as a datetime so the data has the same shape as the ft_notification_status query
-        literal(bst_day.replace(day=1), type_=DateTime).label('month'),
+        literal(fetch_day.replace(day=1), type_=DateTime).label('month'),
         Notification.notification_type,
         Notification.status.label('notification_status'),
         func.count().label('count')
     ).filter(
-        Notification.created_at >= get_london_midnight_in_utc(bst_day),
-        Notification.created_at < get_london_midnight_in_utc(bst_day + timedelta(days=1)),
+        Notification.created_at >= get_midnight_in_utc(fetch_day),
+        Notification.created_at < get_midnight_in_utc(fetch_day + timedelta(days=1)),
         Notification.service_id == service_id,
         Notification.key_type != KEY_TYPE_TEST
     ).group_by(
@@ -141,7 +132,7 @@ def fetch_notification_status_for_service_for_today_and_7_previous_days(service_
         FactNotificationStatus.notification_count.label('count')
     ).filter(
         FactNotificationStatus.service_id == service_id,
-        FactNotificationStatus.bst_date >= start_date,
+        FactNotificationStatus.local_date >= start_date,
         FactNotificationStatus.key_type != KEY_TYPE_TEST
     )
 
@@ -151,7 +142,7 @@ def fetch_notification_status_for_service_for_today_and_7_previous_days(service_
         *([Notification.template_id] if by_template else []),
         func.count().label('count')
     ).filter(
-        Notification.created_at >= get_london_midnight_in_utc(now),
+        Notification.created_at >= get_midnight_in_utc(now),
         Notification.service_id == service_id,
         Notification.key_type != KEY_TYPE_TEST
     ).group_by(
@@ -165,7 +156,7 @@ def fetch_notification_status_for_service_for_today_and_7_previous_days(service_
     query = db.session.query(
         *([
             Template.name.label("template_name"),
-            Template.is_precompiled_letter,
+            False,  # TODO: this is related to is_precompiled_letter
             all_stats_table.c.template_id
         ] if by_template else []),
         all_stats_table.c.notification_type,
@@ -177,7 +168,7 @@ def fetch_notification_status_for_service_for_today_and_7_previous_days(service_
         query = query.filter(all_stats_table.c.template_id == Template.id)
 
     return query.group_by(
-        *([Template.name, Template.is_precompiled_letter, all_stats_table.c.template_id] if by_template else []),
+        *([Template.name, all_stats_table.c.template_id] if by_template else []),
         all_stats_table.c.notification_type,
         all_stats_table.c.status,
     ).all()
@@ -190,14 +181,14 @@ def fetch_notification_status_totals_for_all_services(start_date, end_date):
         FactNotificationStatus.key_type.label('key_type'),
         func.sum(FactNotificationStatus.notification_count).label('count')
     ).filter(
-        FactNotificationStatus.bst_date >= start_date,
-        FactNotificationStatus.bst_date <= end_date
+        FactNotificationStatus.local_date >= start_date,
+        FactNotificationStatus.local_date <= end_date
     ).group_by(
         FactNotificationStatus.notification_type,
         FactNotificationStatus.notification_status,
         FactNotificationStatus.key_type,
     )
-    today = get_london_midnight_in_utc(datetime.utcnow())
+    today = get_midnight_in_utc(datetime.utcnow())
     if start_date <= datetime.utcnow().date() <= end_date:
         stats_for_today = db.session.query(
             Notification.notification_type.cast(db.Text).label('notification_type'),
@@ -254,8 +245,8 @@ def fetch_stats_for_all_services_by_date_range(start_date, end_date, include_fro
         FactNotificationStatus.notification_status.label('status'),
         func.sum(FactNotificationStatus.notification_count).label('count')
     ).filter(
-        FactNotificationStatus.bst_date >= start_date,
-        FactNotificationStatus.bst_date <= end_date,
+        FactNotificationStatus.local_date >= start_date,
+        FactNotificationStatus.local_date <= end_date,
         FactNotificationStatus.service_id == Service.id,
     ).group_by(
         FactNotificationStatus.service_id.label('service_id'),
@@ -274,7 +265,7 @@ def fetch_stats_for_all_services_by_date_range(start_date, end_date, include_fro
         stats = stats.filter(FactNotificationStatus.key_type != KEY_TYPE_TEST)
 
     if start_date <= datetime.utcnow().date() <= end_date:
-        today = get_london_midnight_in_utc(datetime.utcnow())
+        today = get_midnight_in_utc(datetime.utcnow())
         subquery = db.session.query(
             Notification.notification_type.cast(db.Text).label('notification_type'),
             Notification.status.label('status'),
@@ -342,40 +333,37 @@ def fetch_monthly_template_usage_for_service(start_date, end_date, service_id):
         FactNotificationStatus.template_id.label('template_id'),
         Template.name.label('name'),
         Template.template_type.label('template_type'),
-        Template.is_precompiled_letter.label('is_precompiled_letter'),
-        extract('month', FactNotificationStatus.bst_date).label('month'),
-        extract('year', FactNotificationStatus.bst_date).label('year'),
+        extract('month', FactNotificationStatus.local_date).label('month'),
+        extract('year', FactNotificationStatus.local_date).label('year'),
         func.sum(FactNotificationStatus.notification_count).label('count')
     ).join(
         Template, FactNotificationStatus.template_id == Template.id
     ).filter(
         FactNotificationStatus.service_id == service_id,
-        FactNotificationStatus.bst_date >= start_date,
-        FactNotificationStatus.bst_date <= end_date,
+        FactNotificationStatus.local_date >= start_date,
+        FactNotificationStatus.local_date <= end_date,
         FactNotificationStatus.key_type != KEY_TYPE_TEST,
         FactNotificationStatus.notification_status != NOTIFICATION_CANCELLED,
     ).group_by(
         FactNotificationStatus.template_id,
         Template.name,
         Template.template_type,
-        Template.is_precompiled_letter,
-        extract('month', FactNotificationStatus.bst_date).label('month'),
-        extract('year', FactNotificationStatus.bst_date).label('year'),
+        extract('month', FactNotificationStatus.local_date).label('month'),
+        extract('year', FactNotificationStatus.local_date).label('year'),
     ).order_by(
-        extract('year', FactNotificationStatus.bst_date),
-        extract('month', FactNotificationStatus.bst_date),
+        extract('year', FactNotificationStatus.local_date),
+        extract('month', FactNotificationStatus.local_date),
         Template.name
     )
 
     if start_date <= datetime.utcnow() <= end_date:
-        today = get_london_midnight_in_utc(datetime.utcnow())
-        month = get_london_month_from_utc_column(Notification.created_at)
+        today = get_midnight_in_utc(datetime.utcnow())
+        month = get_month_from_utc_column(Notification.created_at)
 
         stats_for_today = db.session.query(
             Notification.template_id.label('template_id'),
             Template.name.label('name'),
             Template.template_type.label('template_type'),
-            Template.is_precompiled_letter.label('is_precompiled_letter'),
             extract('month', month).label('month'),
             extract('year', month).label('year'),
             func.count().label('count')
@@ -398,7 +386,6 @@ def fetch_monthly_template_usage_for_service(start_date, end_date, service_id):
         query = db.session.query(
             all_stats_table.c.template_id,
             all_stats_table.c.name,
-            all_stats_table.c.is_precompiled_letter,
             all_stats_table.c.template_type,
             func.cast(all_stats_table.c.month, Integer).label('month'),
             func.cast(all_stats_table.c.year, Integer).label('year'),
@@ -406,7 +393,6 @@ def fetch_monthly_template_usage_for_service(start_date, end_date, service_id):
         ).group_by(
             all_stats_table.c.template_id,
             all_stats_table.c.name,
-            all_stats_table.c.is_precompiled_letter,
             all_stats_table.c.template_type,
             all_stats_table.c.month,
             all_stats_table.c.year,
@@ -422,7 +408,7 @@ def fetch_monthly_template_usage_for_service(start_date, end_date, service_id):
 
 def get_total_notifications_for_date_range(start_date, end_date):
     query = db.session.query(
-        FactNotificationStatus.bst_date.cast(db.Text).label("bst_date"),
+        FactNotificationStatus.local_date.cast(db.Text).label("local_date"),
         func.sum(case(
             [
                 (FactNotificationStatus.notification_type == 'email', FactNotificationStatus.notification_count)
@@ -433,29 +419,24 @@ def get_total_notifications_for_date_range(start_date, end_date):
                 (FactNotificationStatus.notification_type == 'sms', FactNotificationStatus.notification_count)
             ],
             else_=0)).label('sms'),
-        func.sum(case(
-            [
-                (FactNotificationStatus.notification_type == 'letter', FactNotificationStatus.notification_count)
-            ],
-            else_=0)).label('letters'),
     ).filter(
         FactNotificationStatus.key_type != KEY_TYPE_TEST,
     ).group_by(
-        FactNotificationStatus.bst_date
+        FactNotificationStatus.local_date
     ).order_by(
-        FactNotificationStatus.bst_date
+        FactNotificationStatus.local_date
     )
     if start_date and end_date:
         query = query.filter(
-            FactNotificationStatus.bst_date >= start_date,
-            FactNotificationStatus.bst_date <= end_date
+            FactNotificationStatus.local_date >= start_date,
+            FactNotificationStatus.local_date <= end_date
         )
     return query.all()
 
 
 def fetch_monthly_notification_statuses_per_service(start_date, end_date):
     return db.session.query(
-        func.date_trunc('month', FactNotificationStatus.bst_date).cast(Date).label('date_created'),
+        func.date_trunc('month', FactNotificationStatus.local_date).cast(Date).label('date_created'),
         Service.id.label('service_id'),
         Service.name.label('service_name'),
         FactNotificationStatus.notification_type,
@@ -503,15 +484,15 @@ def fetch_monthly_notification_statuses_per_service(start_date, end_date):
         FactNotificationStatus.key_type != KEY_TYPE_TEST,
         Service.research_mode.is_(False),
         Service.restricted.is_(False),
-        FactNotificationStatus.bst_date >= start_date,
-        FactNotificationStatus.bst_date <= end_date,
+        FactNotificationStatus.local_date >= start_date,
+        FactNotificationStatus.local_date <= end_date,
     ).group_by(
         Service.id,
         Service.name,
-        func.date_trunc('month', FactNotificationStatus.bst_date).cast(Date),
+        func.date_trunc('month', FactNotificationStatus.local_date).cast(Date),
         FactNotificationStatus.notification_type,
     ).order_by(
-        func.date_trunc('month', FactNotificationStatus.bst_date).cast(Date),
+        func.date_trunc('month', FactNotificationStatus.local_date).cast(Date),
         Service.id,
         FactNotificationStatus.notification_type,
     ).all()

app/dao/fact_processing_time_dao.py

@@ -18,12 +18,12 @@ def insert_update_processing_time(processing_time):
     '''
     table = FactProcessingTime.__table__
     stmt = insert(table).values(
-        bst_date=processing_time.bst_date,
+        local_date=processing_time.local_date,
         messages_total=processing_time.messages_total,
         messages_within_10_secs=processing_time.messages_within_10_secs
     )
     stmt = stmt.on_conflict_do_update(
-        index_elements=[table.c.bst_date],
+        index_elements=[table.c.local_date],
         set_={
             'messages_total': stmt.excluded.messages_total,
             'messages_within_10_secs': stmt.excluded.messages_within_10_secs,
@@ -35,7 +35,7 @@ def insert_update_processing_time(processing_time):
 
 def get_processing_time_percentage_for_date_range(start_date, end_date):
     query = db.session.query(
-        FactProcessingTime.bst_date.cast(db.Text).label("date"),
+        FactProcessingTime.local_date.cast(db.Text).label("date"),
         FactProcessingTime.messages_total,
         FactProcessingTime.messages_within_10_secs,
         case([
@@ -46,8 +46,8 @@ def get_processing_time_percentage_for_date_range(start_date, end_date):
             (FactProcessingTime.messages_total == 0, 100.0)
         ]).label("percentage")
     ).filter(
-        FactProcessingTime.bst_date >= start_date,
-        FactProcessingTime.bst_date <= end_date
-    ).order_by(FactProcessingTime.bst_date)
+        FactProcessingTime.local_date >= start_date,
+        FactProcessingTime.local_date <= end_date
+    ).order_by(FactProcessingTime.local_date)
 
     return query.all()

app/dao/invited_org_user_dao.py

@@ -1,7 +1,7 @@
 from datetime import datetime, timedelta
 
 from app import db
-from app.models import InvitedOrganisationUser
+from app.models import InvitedOrganizationUser
 
 
 def save_invited_org_user(invited_org_user):
@@ -9,21 +9,21 @@ def save_invited_org_user(invited_org_user):
     db.session.commit()
 
 
-def get_invited_org_user(organisation_id, invited_org_user_id):
-    return InvitedOrganisationUser.query.filter_by(organisation_id=organisation_id, id=invited_org_user_id).one()
+def get_invited_org_user(organization_id, invited_org_user_id):
+    return InvitedOrganizationUser.query.filter_by(organization_id=organization_id, id=invited_org_user_id).one()
 
 
 def get_invited_org_user_by_id(invited_org_user_id):
-    return InvitedOrganisationUser.query.filter_by(id=invited_org_user_id).one()
+    return InvitedOrganizationUser.query.filter_by(id=invited_org_user_id).one()
 
 
-def get_invited_org_users_for_organisation(organisation_id):
-    return InvitedOrganisationUser.query.filter_by(organisation_id=organisation_id).all()
+def get_invited_org_users_for_organization(organization_id):
+    return InvitedOrganizationUser.query.filter_by(organization_id=organization_id).all()
 
 
 def delete_org_invitations_created_more_than_two_days_ago():
-    deleted = db.session.query(InvitedOrganisationUser).filter(
-        InvitedOrganisationUser.created_at <= datetime.utcnow() - timedelta(days=2)
+    deleted = db.session.query(InvitedOrganizationUser).filter(
+        InvitedOrganizationUser.created_at <= datetime.utcnow() - timedelta(days=2)
     ).delete()
     db.session.commit()
     return deleted

app/dao/jobs_dao.py

@@ -2,23 +2,13 @@ import uuid
 from datetime import datetime, timedelta
 
 from flask import current_app
-from notifications_utils.letter_timings import (
-    CANCELLABLE_JOB_LETTER_STATUSES,
-    letter_can_be_cancelled,
-)
 from sqlalchemy import and_, asc, desc, func
 
 from app import db
-from app.dao.dao_utils import autocommit
-from app.dao.templates_dao import dao_get_template_by_id
 from app.models import (
-    JOB_STATUS_CANCELLED,
     JOB_STATUS_FINISHED,
     JOB_STATUS_PENDING,
     JOB_STATUS_SCHEDULED,
-    LETTER_TYPE,
-    NOTIFICATION_CANCELLED,
-    NOTIFICATION_CREATED,
     FactNotificationStatus,
     Job,
     Notification,
@@ -53,6 +43,10 @@ def dao_get_job_by_service_id_and_job_id(service_id, job_id):
     return Job.query.filter_by(service_id=service_id, id=job_id).one()
 
 
+def dao_get_unfinished_jobs():
+    return Job.query.filter(Job.processing_finished.is_(None)).all()
+
+
 def dao_get_jobs_by_service_id(
     service_id,
     *,
@@ -60,7 +54,6 @@ def dao_get_jobs_by_service_id(
     page=1,
     page_size=50,
     statuses=None,
-    contact_list_id=None,
 ):
     query_filter = [
         Job.service_id == service_id,
@@ -73,8 +66,6 @@ def dao_get_jobs_by_service_id(
         query_filter.append(
             Job.job_status.in_(statuses)
         )
-    if contact_list_id is not None:
-        query_filter.append(Job.contact_list_id == contact_list_id)
     return Job.query \
         .filter(*query_filter) \
         .order_by(Job.processing_started.desc(), Job.created_at.desc()) \
@@ -183,40 +174,6 @@ def dao_get_jobs_older_than_data_retention(notification_types):
     return jobs
 
 
-@autocommit
-def dao_cancel_letter_job(job):
-    number_of_notifications_cancelled = Notification.query.filter(
-        Notification.job_id == job.id
-    ).update({'status': NOTIFICATION_CANCELLED,
-              'updated_at': datetime.utcnow(),
-              'billable_units': 0})
-    job.job_status = JOB_STATUS_CANCELLED
-    dao_update_job(job)
-    return number_of_notifications_cancelled
-
-
-def can_letter_job_be_cancelled(job):
-    template = dao_get_template_by_id(job.template_id)
-    if template.template_type != LETTER_TYPE:
-        return False, "Only letter jobs can be cancelled through this endpoint. This is not a letter job."
-
-    notifications = Notification.query.filter(
-        Notification.job_id == job.id
-    ).all()
-    count_notifications = len(notifications)
-    if job.job_status != JOB_STATUS_FINISHED or count_notifications != job.notification_count:
-        return False, "We are still processing these letters, please try again in a minute."
-    count_cancellable_notifications = len([
-        n for n in notifications if n.status in CANCELLABLE_JOB_LETTER_STATUSES
-    ])
-    if count_cancellable_notifications != job.notification_count or not letter_can_be_cancelled(
-        NOTIFICATION_CREATED, job.created_at
-    ):
-        return False, "It’s too late to cancel sending, these letters have already been sent."
-
-    return True, None
-
-
 def find_jobs_with_missing_rows():
     # Jobs can be a maximum of 100,000 rows. It typically takes 10 minutes to create all those notifications.
     # Using 20 minutes as a condition seems reasonable.

app/dao/letter_branding_dao.py

@@ -1,29 +0,0 @@
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import LetterBranding
-
-
-def dao_get_letter_branding_by_id(letter_branding_id):
-    return LetterBranding.query.filter(LetterBranding.id == letter_branding_id).one()
-
-
-def dao_get_letter_branding_by_name(letter_branding_name):
-    return LetterBranding.query.filter_by(name=letter_branding_name).first()
-
-
-def dao_get_all_letter_branding():
-    return LetterBranding.query.order_by(LetterBranding.name).all()
-
-
-@autocommit
-def dao_create_letter_branding(letter_branding):
-    db.session.add(letter_branding)
-
-
-@autocommit
-def dao_update_letter_branding(letter_branding_id, **kwargs):
-    letter_branding = LetterBranding.query.get(letter_branding_id)
-    for key, value in kwargs.items():
-        setattr(letter_branding, key, value or None)
-    db.session.add(letter_branding)
-    return letter_branding

app/dao/notifications_dao.py

@@ -1,8 +1,5 @@
 from datetime import datetime, timedelta
-from itertools import groupby
-from operator import attrgetter
 
-from botocore.exceptions import ClientError
 from flask import current_app
 from notifications_utils.international_billing_rates import (
     INTERNATIONAL_BILLING_RATES,
@@ -12,43 +9,33 @@ from notifications_utils.recipients import (
     try_validate_and_format_phone_number,
     validate_and_format_email_address,
 )
-from notifications_utils.timezones import convert_bst_to_utc, convert_utc_to_bst
-from sqlalchemy import and_, asc, desc, func, or_
+from sqlalchemy import asc, desc, func, or_, union
 from sqlalchemy.orm import joinedload
 from sqlalchemy.orm.exc import NoResultFound
 from sqlalchemy.sql import functions
 from sqlalchemy.sql.expression import case
 from werkzeug.datastructures import MultiDict
 
-from app import create_uuid, db, statsd_client
-from app.clients.sms.firetext import (
-    get_message_status_and_reason_from_firetext_code,
-)
+from app import create_uuid, db
 from app.dao.dao_utils import autocommit
-from app.letters.utils import LetterPDFNotFound, find_letter_pdf_in_s3
 from app.models import (
     EMAIL_TYPE,
-    KEY_TYPE_NORMAL,
     KEY_TYPE_TEST,
-    LETTER_TYPE,
     NOTIFICATION_CREATED,
-    NOTIFICATION_DELIVERED,
     NOTIFICATION_PENDING,
     NOTIFICATION_PENDING_VIRUS_CHECK,
     NOTIFICATION_PERMANENT_FAILURE,
     NOTIFICATION_SENDING,
     NOTIFICATION_SENT,
-    NOTIFICATION_STATUS_TYPES_COMPLETED,
     NOTIFICATION_TEMPORARY_FAILURE,
     SMS_TYPE,
     FactNotificationStatus,
     Notification,
     NotificationHistory,
-    ProviderDetails,
 )
 from app.utils import (
     escape_special_characters,
-    get_london_midnight_in_utc,
+    get_midnight_in_utc,
     midnight_n_days_ago,
 )
 
@@ -66,7 +53,7 @@ def dao_get_last_date_template_was_used(template_id, service_id):
         return last_date_from_notifications
 
     last_date = db.session.query(
-        functions.max(FactNotificationStatus.bst_date)
+        functions.max(FactNotificationStatus.local_date)
     ).filter(
         FactNotificationStatus.template_id == template_id,
         FactNotificationStatus.key_type != KEY_TYPE_TEST
@@ -86,41 +73,29 @@ def dao_create_notification(notification):
     db.session.add(notification)
 
 
-def _decide_permanent_temporary_failure(status, notification, detailed_status_code=None):
-    # Firetext will send us a pending status, followed by a success or failure status.
-    # When we get a failure status we need to look at the detailed_status_code to determine if the failure type
-    # is a permanent-failure or temporary-failure.
-    if notification.sent_by == 'firetext':
-        if status == NOTIFICATION_PERMANENT_FAILURE and detailed_status_code:
-            try:
-                status, reason = get_message_status_and_reason_from_firetext_code(detailed_status_code)
-                current_app.logger.info(
-                    f'Updating notification id {notification.id} to status {status}, reason: {reason}')
-                return status
-            except KeyError:
-                current_app.logger.warning(f'Failure code {detailed_status_code} from Firetext not recognised')
-        # fallback option:
-        if status == NOTIFICATION_PERMANENT_FAILURE and notification.status == NOTIFICATION_PENDING:
-            status = NOTIFICATION_TEMPORARY_FAILURE
-    return status
-
-
 def country_records_delivery(phone_prefix):
     dlr = INTERNATIONAL_BILLING_RATES[phone_prefix]['attributes']['dlr']
     return dlr and dlr.lower() == 'yes'
 
 
-def _update_notification_status(notification, status, detailed_status_code=None):
-    status = _decide_permanent_temporary_failure(
-        status=status, notification=notification, detailed_status_code=detailed_status_code
-    )
+def _decide_permanent_temporary_failure(current_status, status):
+    # If we go from pending to delivered we need to set failure type as temporary-failure
+    if current_status == NOTIFICATION_PENDING and status == NOTIFICATION_PERMANENT_FAILURE:
+        status = NOTIFICATION_TEMPORARY_FAILURE
+    return status
+
+
+def _update_notification_status(notification, status, provider_response=None):
+    status = _decide_permanent_temporary_failure(current_status=notification.status, status=status)
     notification.status = status
+    if provider_response:
+        notification.provider_response = provider_response
     dao_update_notification(notification)
     return notification
 
 
 @autocommit
-def update_notification_status_by_id(notification_id, status, sent_by=None, detailed_status_code=None):
+def update_notification_status_by_id(notification_id, status, sent_by=None, provider_response=None):
     notification = Notification.query.with_for_update().filter(Notification.id == notification_id).first()
 
     if not notification:
@@ -146,18 +121,19 @@ def update_notification_status_by_id(notification_id, status, sent_by=None, deta
         and not country_records_delivery(notification.phone_prefix)
     ):
         return None
+    if provider_response:
+        notification.provider_response = provider_response
     if not notification.sent_by and sent_by:
         notification.sent_by = sent_by
     return _update_notification_status(
         notification=notification,
-        status=status,
-        detailed_status_code=detailed_status_code
+        status=status
     )
 
 
 @autocommit
 def update_notification_status_by_reference(reference, status):
-    # this is used to update letters and emails
+    # this is used to update emails
     notification = Notification.query.filter(Notification.reference == reference).first()
 
     if not notification:
@@ -283,8 +259,8 @@ def _filter_query(query, filter_dict=None):
 
     # filter by status
     statuses = multidict.getlist('status')
+
     if statuses:
-        statuses = Notification.substitute_status(statuses)
         query = query.filter(Notification.status.in_(statuses))
 
     # filter by template
@@ -295,6 +271,30 @@ def _filter_query(query, filter_dict=None):
     return query
 
 
+@autocommit
+def sanitize_successful_notification_by_id(
+    notification_id
+):
+    # TODO what to do for international?
+    # phone_prefix = '1'
+    # Notification.query.filter(
+    #     Notification.id.in_([notification_id]),
+    # ).update(
+    #     {'to': phone_prefix, 'normalised_to': phone_prefix, 'status': 'delivered'}
+    # )
+    # db.session.commit()
+
+    update_query = """
+    update notifications set notification_status='delivered', "to"='1', normalised_to='1'
+    where id=:notification_id
+    """
+    input_params = {
+        "notification_id": notification_id
+    }
+
+    db.session.execute(update_query, input_params)
+
+
 @autocommit
 def insert_notification_history_delete_notifications(
     notification_type, service_id, timestamp_to_delete_backwards_from, qry_limit=50000
@@ -321,7 +321,7 @@ def insert_notification_history_delete_notifications(
          SELECT id, job_id, job_row_number, service_id, template_id, template_version, api_key_id,
              key_type, notification_type, created_at, sent_at, sent_by, updated_at, reference, billable_units,
              client_reference, international, phone_prefix, rate_multiplier, notification_status,
-              created_by_id, postage, document_download_count
+              created_by_id, document_download_count
           FROM notifications
         WHERE service_id = :service_id
           AND notification_type = :notification_type
@@ -329,20 +329,6 @@ def insert_notification_history_delete_notifications(
           AND key_type in ('normal', 'team')
         limit :qry_limit
         """
-    select_into_temp_table_for_letters = """
-         CREATE TEMP TABLE NOTIFICATION_ARCHIVE ON COMMIT DROP AS
-         SELECT id, job_id, job_row_number, service_id, template_id, template_version, api_key_id,
-             key_type, notification_type, created_at, sent_at, sent_by, updated_at, reference, billable_units,
-             client_reference, international, phone_prefix, rate_multiplier, notification_status,
-              created_by_id, postage, document_download_count
-          FROM notifications
-        WHERE service_id = :service_id
-          AND notification_type = :notification_type
-          AND created_at < :timestamp_to_delete_backwards_from
-          AND notification_status NOT IN ('pending-virus-check', 'created', 'sending')
-          AND key_type in ('normal', 'team')
-        limit :qry_limit
-        """
     # Insert into NotificationHistory if the row already exists do nothing.
     insert_query = """
         insert into notification_history
@@ -361,8 +347,7 @@ def insert_notification_history_delete_notifications(
         "qry_limit": qry_limit
     }
 
-    select_to_use = select_into_temp_table_for_letters if notification_type == 'letter' else select_into_temp_table
-    db.session.execute(select_to_use, input_params)
+    db.session.execute(select_into_temp_table, input_params)
 
     result = db.session.execute("select count(*) from NOTIFICATION_ARCHIVE").fetchone()[0]
 
@@ -380,10 +365,6 @@ def move_notifications_to_notification_history(
     qry_limit=50000
 ):
     deleted = 0
-    if notification_type == LETTER_TYPE:
-        _delete_letters_from_s3(
-            notification_type, service_id, timestamp_to_delete_backwards_from, qry_limit
-        )
     delete_count_per_call = 1
     while delete_count_per_call > 0:
         delete_count_per_call = insert_notification_history_delete_notifications(
@@ -406,32 +387,6 @@ def move_notifications_to_notification_history(
     return deleted
 
 
-def _delete_letters_from_s3(
-        notification_type, service_id, date_to_delete_from, query_limit
-):
-    letters_to_delete_from_s3 = db.session.query(
-        Notification
-    ).filter(
-        Notification.notification_type == notification_type,
-        Notification.created_at < date_to_delete_from,
-        Notification.service_id == service_id,
-        # although letters in non completed statuses do have PDFs in s3, they do not exist in the
-        # production-letters-pdf bucket as they never made it that far so we do not try and delete
-        # them from it
-        Notification.status.in_(NOTIFICATION_STATUS_TYPES_COMPLETED)
-    ).limit(query_limit).all()
-    for letter in letters_to_delete_from_s3:
-        try:
-            letter_pdf = find_letter_pdf_in_s3(letter)
-            letter_pdf.delete()
-        except ClientError:
-            current_app.logger.exception(
-                "Error deleting S3 object for letter: {}".format(letter.id))
-        except LetterPDFNotFound:
-            current_app.logger.warning(
-                "No S3 object to delete for letter: {}".format(letter.id))
-
-
 @autocommit
 def dao_delete_notifications_by_id(notification_id):
     db.session.query(Notification).filter(
@@ -465,61 +420,6 @@ def dao_timeout_notifications(cutoff_time, limit=100000):
     return notifications
 
 
-def is_delivery_slow_for_providers(
-        created_at,
-        threshold,
-        delivery_time,
-):
-    """
-    Returns a dict of providers and whether they are currently slow or not. eg:
-    {
-        'mmg': True,
-        'firetext': False
-    }
-    """
-    slow_notification_counts = db.session.query(
-        ProviderDetails.identifier,
-        case(
-            [(
-                Notification.status == NOTIFICATION_DELIVERED,
-                (Notification.updated_at - Notification.sent_at) >= delivery_time
-            )],
-            else_=(datetime.utcnow() - Notification.sent_at) >= delivery_time
-        ).label("slow"),
-        func.count().label('count')
-    ).select_from(
-        ProviderDetails
-    ).outerjoin(
-        Notification, and_(
-            Notification.notification_type == SMS_TYPE,
-            Notification.sent_by == ProviderDetails.identifier,
-            Notification.created_at >= created_at,
-            Notification.sent_at.isnot(None),
-            Notification.status.in_([NOTIFICATION_DELIVERED, NOTIFICATION_PENDING, NOTIFICATION_SENDING]),
-            Notification.key_type != KEY_TYPE_TEST
-        )
-    ).filter(
-        ProviderDetails.notification_type == 'sms',
-        ProviderDetails.active
-    ).order_by(
-        ProviderDetails.identifier
-    ).group_by(
-        ProviderDetails.identifier,
-        "slow"
-    )
-
-    slow_providers = {}
-    for provider, rows in groupby(slow_notification_counts, key=attrgetter('identifier')):
-        rows = list(rows)
-        total_notifications = sum(row.count for row in rows)
-        slow_notifications = sum(row.count for row in rows if row.slow)
-
-        slow_providers[provider] = (slow_notifications / total_notifications >= threshold)
-        statsd_client.gauge(f'slow-delivery.{provider}.ratio', slow_notifications / total_notifications)
-
-    return slow_providers
-
-
 @autocommit
 def dao_update_notifications_by_reference(references, update_dict):
     updated_count = Notification.query.filter(
@@ -565,10 +465,8 @@ def dao_get_notifications_by_recipient_or_reference(
         except InvalidEmailError:
             normalised = search_term.lower()
 
-    elif notification_type in {LETTER_TYPE, None}:
-        # For letters, we store the address without spaces, so we need
-        # to removes spaces from the search term to match. We also do
-        # this when a notification type isn’t provided (this will
+    elif notification_type is None:
+        # This happens when a notification type isn’t provided (this will
         # happen if a user doesn’t have permission to see the dashboard)
         # because email addresses and phone numbers will never be stored
         # with spaces either.
@@ -576,7 +474,7 @@ def dao_get_notifications_by_recipient_or_reference(
 
     else:
         raise TypeError(
-            f'Notification type must be {EMAIL_TYPE}, {SMS_TYPE}, {LETTER_TYPE} or None'
+            f'Notification type must be {EMAIL_TYPE}, {SMS_TYPE}, or None'
         )
 
     normalised = escape_special_characters(normalised)
@@ -609,17 +507,13 @@ def dao_get_notification_by_reference(reference):
     ).one()
 
 
-def dao_get_notification_or_history_by_reference(reference):
+def dao_get_notification_history_by_reference(reference):
     try:
         # This try except is necessary because in test keys and research mode does not create notification history.
         # Otherwise we could just search for the NotificationHistory object
-        return Notification.query.filter(
-            Notification.reference == reference
-        ).one()
+        return Notification.query.filter(Notification.reference == reference).one()
     except NoResultFound:
-        return NotificationHistory.query.filter(
-            NotificationHistory.reference == reference
-        ).one()
+        return NotificationHistory.query.filter(NotificationHistory.reference == reference).one()
 
 
 def dao_get_notifications_processing_time_stats(start_date, end_date):
@@ -635,8 +529,7 @@ def dao_get_notifications_processing_time_stats(start_date, end_date):
     created_at > 'START DATE' AND
     created_at < 'END DATE' AND
     api_key_id IS NOT NULL AND
-    key_type != 'test' AND
-    notification_type != 'letter';
+    key_type != 'test';
     """
     under_10_secs = Notification.sent_at - Notification.created_at <= timedelta(seconds=10)
     sum_column = functions.coalesce(functions.sum(
@@ -656,7 +549,6 @@ def dao_get_notifications_processing_time_stats(start_date, end_date):
         Notification.created_at < end_date,
         Notification.api_key_id.isnot(None),
         Notification.key_type != KEY_TYPE_TEST,
-        Notification.notification_type != LETTER_TYPE
     ).one()
 
 
@@ -681,97 +573,6 @@ def notifications_not_yet_sent(should_be_sending_after_seconds, notification_typ
     return notifications
 
 
-def dao_get_letters_to_be_printed(print_run_deadline, postage, query_limit=10000):
-    """
-    Return all letters created before the print run deadline that have not yet been sent. This yields in batches of 10k
-    to prevent the query taking too long and eating up too much memory. As each 10k batch is yielded, the
-    get_key_and_size_of_letters_to_be_sent_to_print function will go and fetch the s3 data, andhese  start sending off
-    tasks to the notify-ftp app to send them.
-
-    CAUTION! Modify this query with caution. Modifying filters etc is fine, but if we join onto another table, then
-    there may be undefined behaviour. Essentially we need each ORM object returned for each row to be unique,
-    and we should avoid modifying state of returned objects.
-
-    For more reading:
-    https://docs.sqlalchemy.org/en/13/orm/query.html?highlight=yield_per#sqlalchemy.orm.query.Query.yield_per
-    https://www.mail-archive.com/sqlalchemy@googlegroups.com/msg12443.html
-    """
-    notifications = Notification.query.filter(
-        Notification.created_at < convert_bst_to_utc(print_run_deadline),
-        Notification.notification_type == LETTER_TYPE,
-        Notification.status == NOTIFICATION_CREATED,
-        Notification.key_type == KEY_TYPE_NORMAL,
-        Notification.postage == postage,
-        Notification.billable_units > 0
-    ).order_by(
-        Notification.service_id,
-        Notification.created_at
-    ).yield_per(query_limit)
-    return notifications
-
-
-def dao_get_letters_and_sheets_volume_by_postage(print_run_deadline):
-    notifications = db.session.query(
-        func.count(Notification.id).label('letters_count'),
-        func.sum(Notification.billable_units).label('sheets_count'),
-        Notification.postage
-    ).filter(
-        Notification.created_at < convert_bst_to_utc(print_run_deadline),
-        Notification.notification_type == LETTER_TYPE,
-        Notification.status == NOTIFICATION_CREATED,
-        Notification.key_type == KEY_TYPE_NORMAL,
-        Notification.billable_units > 0
-    ).group_by(
-        Notification.postage
-    ).order_by(
-        Notification.postage
-    ).all()
-    return notifications
-
-
-def dao_old_letters_with_created_status():
-    yesterday_bst = convert_utc_to_bst(datetime.utcnow()) - timedelta(days=1)
-    last_processing_deadline = yesterday_bst.replace(hour=17, minute=30, second=0, microsecond=0)
-
-    notifications = Notification.query.filter(
-        Notification.created_at < convert_bst_to_utc(last_processing_deadline),
-        Notification.notification_type == LETTER_TYPE,
-        Notification.status == NOTIFICATION_CREATED
-    ).order_by(
-        Notification.created_at
-    ).all()
-    return notifications
-
-
-def letters_missing_from_sending_bucket(seconds_to_subtract):
-    older_than_date = datetime.utcnow() - timedelta(seconds=seconds_to_subtract)
-    # We expect letters to have a `created` status, updated_at timestamp and billable units greater than zero.
-    notifications = Notification.query.filter(
-        Notification.billable_units == 0,
-        Notification.updated_at == None,  # noqa
-        Notification.status == NOTIFICATION_CREATED,
-        Notification.created_at <= older_than_date,
-        Notification.notification_type == LETTER_TYPE,
-        Notification.key_type == KEY_TYPE_NORMAL
-    ).order_by(
-        Notification.created_at
-    ).all()
-
-    return notifications
-
-
-def dao_precompiled_letters_still_pending_virus_check():
-    ninety_minutes_ago = datetime.utcnow() - timedelta(seconds=5400)
-
-    notifications = Notification.query.filter(
-        Notification.created_at < ninety_minutes_ago,
-        Notification.status == NOTIFICATION_PENDING_VIRUS_CHECK
-    ).order_by(
-        Notification.created_at
-    ).all()
-    return notifications
-
-
 def _duplicate_update_warning(notification, status):
     current_app.logger.info(
         (
@@ -804,17 +605,29 @@ def get_service_ids_with_notifications_before(notification_type, timestamp):
 
 
 def get_service_ids_with_notifications_on_date(notification_type, date):
-    start_date = get_london_midnight_in_utc(date)
-    end_date = get_london_midnight_in_utc(date + timedelta(days=1))
+    start_date = get_midnight_in_utc(date)
+    end_date = get_midnight_in_utc(date + timedelta(days=1))
+
+    notification_table_query = db.session.query(
+        Notification.service_id.label('service_id')
+    ).filter(
+        Notification.notification_type == notification_type,
+        # using >= + < is much more efficient than date(created_at)
+        Notification.created_at >= start_date,
+        Notification.created_at < end_date,
+    )
+
+    # Looking at this table is more efficient for historical notifications,
+    # provided the task to populate it has run before they were archived.
+    ft_status_table_query = db.session.query(
+        FactNotificationStatus.service_id.label('service_id')
+    ).filter(
+        FactNotificationStatus.notification_type == notification_type,
+        FactNotificationStatus.local_date == date,
+    )
 
     return {
-        row.service_id
-        for row in db.session.query(
-            Notification.service_id
-        ).filter(
-            Notification.notification_type == notification_type,
-            # using >= + < is much more efficient than date(created_at)
-            Notification.created_at >= start_date,
-            Notification.created_at < end_date,
-        ).distinct()
+        row.service_id for row in db.session.query(union(
+            notification_table_query, ft_status_table_query
+        ).subquery()).distinct()
     }

app/dao/organisation_dao.py

@@ -1,144 +0,0 @@
-from sqlalchemy.sql.expression import func
-
-from app import db
-from app.dao.dao_utils import VersionOptions, autocommit, version_class
-from app.models import Domain, Organisation, Service, User
-
-
-def dao_get_organisations():
-    return Organisation.query.order_by(
-        Organisation.active.desc(), Organisation.name.asc()
-    ).all()
-
-
-def dao_count_organisations_with_live_services():
-    return db.session.query(Organisation.id).join(Organisation.services).filter(
-        Service.active.is_(True),
-        Service.restricted.is_(False),
-        Service.count_as_live.is_(True),
-    ).distinct().count()
-
-
-def dao_get_organisation_services(organisation_id):
-    return Organisation.query.filter_by(
-        id=organisation_id
-    ).one().services
-
-
-def dao_get_organisation_live_services(organisation_id):
-    return Service.query.filter_by(
-        organisation_id=organisation_id,
-        restricted=False
-    ).all()
-
-
-def dao_get_organisation_by_id(organisation_id):
-    return Organisation.query.filter_by(id=organisation_id).one()
-
-
-def dao_get_organisation_by_email_address(email_address):
-
-    email_address = email_address.lower().replace('.gsi.gov.uk', '.gov.uk')
-
-    for domain in Domain.query.order_by(func.char_length(Domain.domain).desc()).all():
-
-        if (
-            email_address.endswith("@{}".format(domain.domain)) or
-            email_address.endswith(".{}".format(domain.domain))
-        ):
-            return Organisation.query.filter_by(id=domain.organisation_id).one()
-
-    return None
-
-
-def dao_get_organisation_by_service_id(service_id):
-    return Organisation.query.join(Organisation.services).filter_by(id=service_id).first()
-
-
-@autocommit
-def dao_create_organisation(organisation):
-    db.session.add(organisation)
-
-
-@autocommit
-def dao_update_organisation(organisation_id, **kwargs):
-
-    domains = kwargs.pop('domains', None)
-
-    num_updated = Organisation.query.filter_by(id=organisation_id).update(
-        kwargs
-    )
-
-    if isinstance(domains, list):
-
-        Domain.query.filter_by(organisation_id=organisation_id).delete()
-
-        db.session.bulk_save_objects([
-            Domain(domain=domain.lower(), organisation_id=organisation_id)
-            for domain in domains
-        ])
-
-    organisation = Organisation.query.get(organisation_id)
-
-    if 'organisation_type' in kwargs:
-        _update_organisation_services(organisation, 'organisation_type', only_where_none=False)
-
-    if 'crown' in kwargs:
-        _update_organisation_services(organisation, 'crown', only_where_none=False)
-
-    if 'email_branding_id' in kwargs:
-        _update_organisation_services(organisation, 'email_branding')
-
-    if 'letter_branding_id' in kwargs:
-        _update_organisation_services(organisation, 'letter_branding')
-
-    return num_updated
-
-
-@version_class(
-    VersionOptions(Service, must_write_history=False),
-)
-def _update_organisation_services(organisation, attribute, only_where_none=True):
-    for service in organisation.services:
-        if getattr(service, attribute) is None or not only_where_none:
-            setattr(service, attribute, getattr(organisation, attribute))
-        db.session.add(service)
-
-
-@autocommit
-@version_class(Service)
-def dao_add_service_to_organisation(service, organisation_id):
-    organisation = Organisation.query.filter_by(
-        id=organisation_id
-    ).one()
-
-    service.organisation_id = organisation_id
-    service.organisation_type = organisation.organisation_type
-    service.crown = organisation.crown
-
-    db.session.add(service)
-
-
-def dao_get_users_for_organisation(organisation_id):
-    return db.session.query(
-        User
-    ).join(
-        User.organisations
-    ).filter(
-        Organisation.id == organisation_id,
-        User.state == 'active'
-    ).order_by(User.created_at).all()
-
-
-@autocommit
-def dao_add_user_to_organisation(organisation_id, user_id):
-    organisation = dao_get_organisation_by_id(organisation_id)
-    user = User.query.filter_by(id=user_id).one()
-    user.organisations.append(organisation)
-    db.session.add(organisation)
-    return user
-
-
-@autocommit
-def dao_remove_user_from_organisation(organisation, user):
-    organisation.users.remove(user)

app/dao/organization_dao.py

@@ -0,0 +1,132 @@
+from sqlalchemy.sql.expression import func
+
+from app import db
+from app.dao.dao_utils import VersionOptions, autocommit, version_class
+from app.models import Domain, Organization, Service, User
+
+
+def dao_get_organizations():
+    return Organization.query.order_by(
+        Organization.active.desc(), Organization.name.asc()
+    ).all()
+
+
+def dao_count_organizations_with_live_services():
+    return db.session.query(Organization.id).join(Organization.services).filter(
+        Service.active.is_(True),
+        Service.restricted.is_(False),
+        Service.count_as_live.is_(True),
+    ).distinct().count()
+
+
+def dao_get_organization_services(organization_id):
+    return Organization.query.filter_by(
+        id=organization_id
+    ).one().services
+
+
+def dao_get_organization_live_services(organization_id):
+    return Service.query.filter_by(
+        organization_id=organization_id,
+        restricted=False
+    ).all()
+
+
+def dao_get_organization_by_id(organization_id):
+    return Organization.query.filter_by(id=organization_id).one()
+
+
+def dao_get_organization_by_email_address(email_address):
+
+    email_address = email_address.lower().replace('.gsi.gov.uk', '.gov.uk')
+
+    for domain in Domain.query.order_by(func.char_length(Domain.domain).desc()).all():
+
+        if (
+            email_address.endswith("@{}".format(domain.domain)) or
+            email_address.endswith(".{}".format(domain.domain))
+        ):
+            return Organization.query.filter_by(id=domain.organization_id).one()
+
+    return None
+
+
+def dao_get_organization_by_service_id(service_id):
+    return Organization.query.join(Organization.services).filter_by(id=service_id).first()
+
+
+@autocommit
+def dao_create_organization(organization):
+    db.session.add(organization)
+
+
+@autocommit
+def dao_update_organization(organization_id, **kwargs):
+    domains = kwargs.pop('domains', None)
+    num_updated = Organization.query.filter_by(id=organization_id).update(
+        kwargs
+    )
+
+    if isinstance(domains, list):
+        Domain.query.filter_by(organization_id=organization_id).delete()
+        db.session.bulk_save_objects([
+            Domain(domain=domain.lower(), organization_id=organization_id)
+            for domain in domains
+        ])
+
+    organization = Organization.query.get(organization_id)
+    if 'organization_type' in kwargs:
+        _update_organization_services(organization, 'organization_type', only_where_none=False)
+
+    if 'email_branding_id' in kwargs:
+        _update_organization_services(organization, 'email_branding')
+
+    return num_updated
+
+
+@version_class(
+    VersionOptions(Service, must_write_history=False),
+)
+def _update_organization_services(organization, attribute, only_where_none=True):
+    for service in organization.services:
+        if getattr(service, attribute) is None or not only_where_none:
+            setattr(service, attribute, getattr(organization, attribute))
+        db.session.add(service)
+
+
+@autocommit
+@version_class(Service)
+def dao_add_service_to_organization(service, organization_id):
+    organization = Organization.query.filter_by(
+        id=organization_id
+    ).one()
+
+    service.organization_id = organization_id
+    service.organization_type = organization.organization_type
+
+    db.session.add(service)
+
+
+def dao_get_users_for_organization(organization_id):
+    return db.session.query(
+        User
+    ).join(
+        User.organizations
+    ).filter(
+        Organization.id == organization_id,
+        User.state == 'active'
+    ).order_by(User.created_at).all()
+
+
+@autocommit
+def dao_add_user_to_organization(organization_id, user_id):
+    organization = dao_get_organization_by_id(organization_id)
+    user = User.query.filter_by(id=user_id).one()
+    user.organizations.append(organization)
+    db.session.add(organization)
+    return user
+
+
+@autocommit
+def dao_remove_user_from_organization(organization, user):
+    organization.users.remove(user)

app/dao/permissions_dao.py

@@ -6,7 +6,6 @@ from app.models import (
     MANAGE_TEMPLATES,
     MANAGE_USERS,
     SEND_EMAILS,
-    SEND_LETTERS,
     SEND_TEXTS,
     VIEW_ACTIVITY,
     Permission,
@@ -19,7 +18,6 @@ default_service_permissions = [
     MANAGE_SETTINGS,
     SEND_TEXTS,
     SEND_EMAILS,
-    SEND_LETTERS,
     MANAGE_API_KEYS,
     VIEW_ACTIVITY]
 

app/dao/provider_details_dao.py

@@ -1,7 +1,6 @@
 from datetime import datetime, timedelta
 
 from flask import current_app
-from notifications_utils.timezones import convert_utc_to_bst
 from sqlalchemy import asc, desc, func
 
 from app import db
@@ -24,10 +23,8 @@ def get_provider_details_by_identifier(identifier):
 
 
 def get_alternative_sms_provider(identifier):
-    if identifier == 'firetext':
-        return 'mmg'
-    elif identifier == 'mmg':
-        return 'firetext'
+    if identifier == 'sns':
+        raise Exception("No alternative SMS providers currently available")
     raise ValueError('Unrecognised sms provider {}'.format(identifier))
 
 
@@ -157,15 +154,15 @@ def _update_provider_details_without_commit(provider_details):
 def dao_get_provider_stats():
     # this query does not include the current day since the task to populate ft_billing runs overnight
 
-    current_bst_datetime = convert_utc_to_bst(datetime.utcnow())
-    first_day_of_the_month = current_bst_datetime.date().replace(day=1)
+    current_datetime = datetime.utcnow()
+    first_day_of_the_month = current_datetime.date().replace(day=1)
 
     subquery = db.session.query(
         FactBilling.provider,
         func.sum(FactBilling.billable_units * FactBilling.rate_multiplier).label('current_month_billable_sms')
     ).filter(
         FactBilling.notification_type == SMS_TYPE,
-        FactBilling.bst_date >= first_day_of_the_month
+        FactBilling.local_date >= first_day_of_the_month
     ).group_by(
         FactBilling.provider
     ).subquery()

app/dao/provider_rates_dao.py

@@ -1,11 +0,0 @@
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import ProviderDetails, ProviderRates
-
-
-@autocommit
-def create_provider_rates(provider_identifier, valid_from, rate):
-    provider = ProviderDetails.query.filter_by(identifier=provider_identifier).one()
-
-    provider_rates = ProviderRates(provider_id=provider.id, valid_from=valid_from, rate=rate)
-    db.session.add(provider_rates)

app/dao/returned_letters_dao.py

@@ -1,118 +0,0 @@
-from datetime import datetime
-
-from sqlalchemy import desc, func
-from sqlalchemy.dialects.postgresql import insert
-
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import (
-    Job,
-    Notification,
-    NotificationHistory,
-    ReturnedLetter,
-    Template,
-    User,
-)
-from app.utils import midnight_n_days_ago
-
-
-def _get_notification_ids_for_references(references):
-    notification_ids = db.session.query(Notification.id, Notification.service_id).filter(
-        Notification.reference.in_(references)
-    ).all()
-
-    notification_history_ids = db.session.query(NotificationHistory.id, NotificationHistory.service_id).filter(
-        NotificationHistory.reference.in_(references)
-    ).all()
-
-    return notification_ids + notification_history_ids
-
-
-@autocommit
-def insert_or_update_returned_letters(references):
-    data = _get_notification_ids_for_references(references)
-    for row in data:
-        table = ReturnedLetter.__table__
-
-        stmt = insert(table).values(
-            reported_at=datetime.utcnow().date(),
-            service_id=row.service_id,
-            notification_id=row.id,
-            created_at=datetime.utcnow()
-        )
-
-        stmt = stmt.on_conflict_do_update(
-            index_elements=[table.c.notification_id],
-            set_={
-                'reported_at': datetime.utcnow().date(),
-                'updated_at': datetime.utcnow()
-            }
-        )
-        db.session.connection().execute(stmt)
-
-
-def fetch_recent_returned_letter_count(service_id):
-    return db.session.query(
-        func.count(ReturnedLetter.notification_id).label('returned_letter_count'),
-    ).filter(
-        ReturnedLetter.service_id == service_id,
-        ReturnedLetter.reported_at > midnight_n_days_ago(7),
-    ).one()
-
-
-def fetch_most_recent_returned_letter(service_id):
-    return db.session.query(
-        ReturnedLetter.reported_at,
-    ).filter(
-        ReturnedLetter.service_id == service_id,
-    ).order_by(
-        desc(ReturnedLetter.reported_at)
-    ).first()
-
-
-def fetch_returned_letter_summary(service_id):
-    return db.session.query(
-        func.count(ReturnedLetter.notification_id).label('returned_letter_count'),
-        ReturnedLetter.reported_at
-    ).filter(
-        ReturnedLetter.service_id == service_id,
-    ).group_by(
-        ReturnedLetter.reported_at
-    ).order_by(
-        desc(ReturnedLetter.reported_at)
-    ).all()
-
-
-def fetch_returned_letters(service_id, report_date):
-    results = []
-    for table in [Notification, NotificationHistory]:
-        query = db.session.query(
-            ReturnedLetter.notification_id,
-            ReturnedLetter.reported_at,
-            table.client_reference,
-            table.created_at,
-            Template.name.label('template_name'),
-            table.template_id,
-            table.template_version,
-            Template.hidden,
-            table.api_key_id,
-            table.created_by_id,
-            User.name.label('user_name'),
-            User.email_address,
-            Job.original_file_name,
-            (table.job_row_number + 1).label('job_row_number')  # row numbers start at 0
-        ).outerjoin(
-            User, table.created_by_id == User.id
-        ).outerjoin(
-            Job, table.job_id == Job.id
-        ).filter(
-            ReturnedLetter.service_id == service_id,
-            ReturnedLetter.reported_at == report_date,
-            ReturnedLetter.notification_id == table.id,
-            table.template_id == Template.id
-        ).order_by(
-            desc(ReturnedLetter.reported_at), desc(table.created_at)
-        )
-        results = results + query.all()
-    results = sorted(results, key=lambda i: i.created_at, reverse=True)
-    return results

app/dao/service_contact_list_dao.py

@@ -1,33 +0,0 @@
-from app import db
-from app.models import ServiceContactList
-
-
-def dao_get_contact_list_by_id(service_id, contact_list_id):
-    contact_list = ServiceContactList.query.filter_by(
-        service_id=service_id,
-        id=contact_list_id,
-        archived=False,
-    ).one()
-
-    return contact_list
-
-
-def dao_get_contact_lists(service_id):
-    contact_lists = ServiceContactList.query.filter_by(
-        service_id=service_id,
-        archived=False,
-    ).order_by(
-        ServiceContactList.created_at.desc()
-    )
-    return contact_lists.all()
-
-
-def save_service_contact_list(service_contact_list):
-    db.session.add(service_contact_list)
-    db.session.commit()
-
-
-def dao_archive_contact_list(service_contact_list):
-    service_contact_list.archived = True
-    db.session.add(service_contact_list)
-    db.session.commit()

app/dao/service_letter_contact_dao.py

@@ -1,105 +0,0 @@
-from sqlalchemy import desc
-
-from app import db
-from app.dao.dao_utils import autocommit
-from app.models import ServiceLetterContact, Template
-
-
-def dao_get_letter_contacts_by_service_id(service_id):
-    letter_contacts = db.session.query(
-        ServiceLetterContact
-    ).filter(
-        ServiceLetterContact.service_id == service_id,
-        ServiceLetterContact.archived == False  # noqa
-    ).order_by(
-        desc(ServiceLetterContact.is_default),
-        desc(ServiceLetterContact.created_at)
-    ).all()
-
-    return letter_contacts
-
-
-def dao_get_letter_contact_by_id(service_id, letter_contact_id):
-    letter_contact = db.session.query(
-        ServiceLetterContact
-    ).filter(
-        ServiceLetterContact.service_id == service_id,
-        ServiceLetterContact.id == letter_contact_id,
-        ServiceLetterContact.archived == False  # noqa
-    ).one()
-    return letter_contact
-
-
-@autocommit
-def add_letter_contact_for_service(service_id, contact_block, is_default):
-    old_default = _get_existing_default(service_id)
-    if is_default:
-        _reset_old_default_to_false(old_default)
-
-    new_letter_contact = ServiceLetterContact(
-        service_id=service_id,
-        contact_block=contact_block,
-        is_default=is_default
-    )
-    db.session.add(new_letter_contact)
-    return new_letter_contact
-
-
-@autocommit
-def update_letter_contact(service_id, letter_contact_id, contact_block, is_default):
-    old_default = _get_existing_default(service_id)
-    # if we want to make this the default, ensure there are no other existing defaults
-    if is_default:
-        _reset_old_default_to_false(old_default)
-
-    letter_contact_update = ServiceLetterContact.query.get(letter_contact_id)
-    letter_contact_update.contact_block = contact_block
-    letter_contact_update.is_default = is_default
-    db.session.add(letter_contact_update)
-    return letter_contact_update
-
-
-@autocommit
-def archive_letter_contact(service_id, letter_contact_id):
-    letter_contact_to_archive = ServiceLetterContact.query.filter_by(
-        id=letter_contact_id,
-        service_id=service_id
-    ).one()
-
-    Template.query.filter_by(
-        service_letter_contact_id=letter_contact_id
-    ).update({
-        'service_letter_contact_id': None
-    })
-
-    letter_contact_to_archive.archived = True
-
-    db.session.add(letter_contact_to_archive)
-    return letter_contact_to_archive
-
-
-def _get_existing_default(service_id):
-    old_defaults = [
-        x for x
-        in dao_get_letter_contacts_by_service_id(service_id=service_id)
-        if x.is_default
-    ]
-
-    if len(old_defaults) == 0:
-        return None
-
-    if len(old_defaults) == 1:
-        return old_defaults[0]
-
-    raise Exception(
-        "There should only be one default letter contact for each service. Service {} has {}".format(
-            service_id,
-            len(old_defaults)
-        )
-    )
-
-
-def _reset_old_default_to_false(old_default):
-    if old_default:
-        old_default.is_default = False
-        db.session.add(old_default)

app/dao/services_dao.py

@@ -1,5 +1,5 @@
 import uuid
-from datetime import date, datetime, timedelta
+from datetime import datetime, timedelta
 
 from flask import current_app
 from sqlalchemy import Float, cast
@@ -8,25 +8,17 @@ from sqlalchemy.sql.expression import and_, asc, case, func
 
 from app import db
 from app.dao.dao_utils import VersionOptions, autocommit, version_class
-from app.dao.date_util import get_current_financial_year
-from app.dao.email_branding_dao import dao_get_email_branding_by_name
-from app.dao.letter_branding_dao import dao_get_letter_branding_by_name
-from app.dao.organisation_dao import dao_get_organisation_by_email_address
+from app.dao.date_util import get_current_calendar_year
+from app.dao.organization_dao import dao_get_organization_by_email_address
 from app.dao.service_sms_sender_dao import insert_service_sms_sender
 from app.dao.service_user_dao import dao_get_service_user
 from app.dao.template_folder_dao import dao_get_valid_template_folders_by_id
 from app.models import (
-    CROWN_ORGANISATION_TYPES,
     EMAIL_TYPE,
-    INTERNATIONAL_LETTERS,
     INTERNATIONAL_SMS_TYPE,
     KEY_TYPE_TEST,
-    LETTER_TYPE,
-    NHS_ORGANISATION_TYPES,
-    NON_CROWN_ORGANISATION_TYPES,
     NOTIFICATION_PERMANENT_FAILURE,
     SMS_TYPE,
-    UPLOAD_LETTERS,
     AnnualBilling,
     ApiKey,
     FactBilling,
@@ -35,12 +27,10 @@ from app.models import (
     Job,
     Notification,
     NotificationHistory,
-    Organisation,
+    Organization,
     Permission,
     Service,
-    ServiceContactList,
     ServiceEmailReplyTo,
-    ServiceLetterContact,
     ServicePermission,
     ServiceSmsSender,
     Template,
@@ -50,20 +40,15 @@ from app.models import (
     VerifyCode,
 )
 from app.utils import (
-    email_address_is_nhs,
     escape_special_characters,
     get_archived_db_column_value,
-    get_london_midnight_in_utc,
-    midnight_n_days_ago,
+    get_midnight_in_utc,
 )
 
 DEFAULT_SERVICE_PERMISSIONS = [
     SMS_TYPE,
     EMAIL_TYPE,
-    LETTER_TYPE,
     INTERNATIONAL_SMS_TYPE,
-    UPLOAD_LETTERS,
-    INTERNATIONAL_LETTERS,
 ]
 
 
@@ -94,7 +79,7 @@ def dao_count_live_services():
 
 
 def dao_fetch_live_services_data():
-    year_start_date, year_end_date = get_current_financial_year()
+    year_start_date, year_end_date = get_current_calendar_year()
 
     most_recent_annual_billing = db.session.query(
         AnnualBilling.service_id,
@@ -104,15 +89,15 @@ def dao_fetch_live_services_data():
     ).subquery()
 
     this_year_ft_billing = FactBilling.query.filter(
-        FactBilling.bst_date >= year_start_date,
-        FactBilling.bst_date <= year_end_date,
+        FactBilling.local_date >= year_start_date,
+        FactBilling.local_date <= year_end_date,
     ).subquery()
 
     data = db.session.query(
         Service.id.label('service_id'),
         Service.name.label("service_name"),
-        Organisation.name.label("organisation_name"),
-        Organisation.organisation_type.label('organisation_type'),
+        Organization.name.label("organization_name"),
+        Organization.organization_type.label('organization_type'),
         Service.consent_to_research.label('consent_to_research'),
         User.name.label('contact_name'),
         User.email_address.label('contact_email'),
@@ -120,16 +105,12 @@ def dao_fetch_live_services_data():
         Service.go_live_at.label("live_date"),
         Service.volume_sms.label('sms_volume_intent'),
         Service.volume_email.label('email_volume_intent'),
-        Service.volume_letter.label('letter_volume_intent'),
         case([
             (this_year_ft_billing.c.notification_type == 'email', func.sum(this_year_ft_billing.c.notifications_sent))
         ], else_=0).label("email_totals"),
         case([
             (this_year_ft_billing.c.notification_type == 'sms', func.sum(this_year_ft_billing.c.notifications_sent))
         ], else_=0).label("sms_totals"),
-        case([
-            (this_year_ft_billing.c.notification_type == 'letter', func.sum(this_year_ft_billing.c.notifications_sent))
-        ], else_=0).label("letter_totals"),
         AnnualBilling.free_sms_fragment_limit,
     ).join(
         Service.annual_billing
@@ -140,7 +121,7 @@ def dao_fetch_live_services_data():
             AnnualBilling.financial_year_start == most_recent_annual_billing.c.year
         )
     ).outerjoin(
-        Service.organisation
+        Service.organization
     ).outerjoin(
         this_year_ft_billing, Service.id == this_year_ft_billing.c.service_id
     ).outerjoin(
@@ -151,8 +132,8 @@ def dao_fetch_live_services_data():
         Service.restricted.is_(False),
     ).group_by(
         Service.id,
-        Organisation.name,
-        Organisation.organisation_type,
+        Organization.name,
+        Organization.organization_type,
         Service.name,
         Service.consent_to_research,
         Service.count_as_live,
@@ -163,7 +144,6 @@ def dao_fetch_live_services_data():
         Service.go_live_at,
         Service.volume_sms,
         Service.volume_email,
-        Service.volume_letter,
         this_year_ft_billing.c.notification_type,
         AnnualBilling.free_sms_fragment_limit,
     ).order_by(
@@ -176,7 +156,6 @@ def dao_fetch_live_services_data():
         if existing_service is not None:
             existing_service["email_totals"] += row.email_totals
             existing_service["sms_totals"] += row.sms_totals
-            existing_service["letter_totals"] += row.letter_totals
         else:
             results.append(row._asdict())
     return results
@@ -299,7 +278,7 @@ def dao_create_service(
     if service_permissions is None:
         service_permissions = DEFAULT_SERVICE_PERMISSIONS
 
-    organisation = dao_get_organisation_by_email_address(user.email_address)
+    organization = dao_get_organization_by_email_address(user.email_address)
 
     from app.dao.permissions_dao import permission_dao
     service.users.append(user)
@@ -315,26 +294,13 @@ def dao_create_service(
     # do we just add the default - or will we get a value from FE?
     insert_service_sms_sender(service, current_app.config['FROM_NUMBER'])
 
-    if organisation:
-        service.organisation_id = organisation.id
-        service.organisation_type = organisation.organisation_type
+    if organization:
+        service.organization_id = organization.id
+        service.organization_type = organization.organization_type
 
-        if organisation.email_branding:
-            service.email_branding = organisation.email_branding
+        if organization.email_branding:
+            service.email_branding = organization.email_branding
 
-        if organisation.letter_branding:
-            service.letter_branding = organisation.letter_branding
-
-    elif service.organisation_type in NHS_ORGANISATION_TYPES or email_address_is_nhs(user.email_address):
-        service.email_branding = dao_get_email_branding_by_name('NHS')
-        service.letter_branding = dao_get_letter_branding_by_name('NHS')
-
-    if organisation:
-        service.crown = organisation.crown
-    elif service.organisation_type in CROWN_ORGANISATION_TYPES:
-        service.crown = True
-    elif service.organisation_type in NON_CROWN_ORGANISATION_TYPES:
-        service.crown = False
     service.count_as_live = not user.platform_admin
 
     db.session.add(service)
@@ -393,8 +359,6 @@ def delete_service_and_all_associated_db_objects(service):
 
     _delete_commit(ServiceSmsSender.query.filter_by(service=service))
     _delete_commit(ServiceEmailReplyTo.query.filter_by(service=service))
-    _delete_commit(ServiceLetterContact.query.filter_by(service=service))
-    _delete_commit(ServiceContactList.query.filter_by(service=service))
     _delete_commit(InvitedUser.query.filter_by(service=service))
     _delete_commit(Permission.query.filter_by(service=service))
     _delete_commit(NotificationHistory.query.filter_by(service=service))
@@ -412,7 +376,7 @@ def delete_service_and_all_associated_db_objects(service):
     db.session.commit()
     users = [x for x in service.users]
     for user in users:
-        user.organisations = []
+        user.organizations = []
         service.users.remove(user)
     _delete_commit(Service.get_history_model().query.filter_by(id=service.id))
     db.session.delete(service)
@@ -422,40 +386,27 @@ def delete_service_and_all_associated_db_objects(service):
     db.session.commit()
 
 
-def dao_fetch_stats_for_service(service_id, limit_days):
-    # We always want between seven and eight days
-    start_date = midnight_n_days_ago(limit_days)
-    return _stats_for_service_query(service_id).filter(
-        Notification.created_at >= start_date
-    ).all()
-
-
 def dao_fetch_todays_stats_for_service(service_id):
-    today = date.today()
-    start_date = get_london_midnight_in_utc(today)
-    return _stats_for_service_query(service_id).filter(
-        Notification.created_at >= start_date
-    ).all()
-
-
-def _stats_for_service_query(service_id):
+    today = datetime.utcnow().date()
+    start_date = get_midnight_in_utc(today)
     return db.session.query(
         Notification.notification_type,
         Notification.status,
         func.count(Notification.id).label('count')
     ).filter(
         Notification.service_id == service_id,
-        Notification.key_type != KEY_TYPE_TEST
+        Notification.key_type != KEY_TYPE_TEST,
+        Notification.created_at >= start_date
     ).group_by(
         Notification.notification_type,
         Notification.status,
-    )
+    ).all()
 
 
 def dao_fetch_todays_stats_for_all_services(include_from_test_key=True, only_active=True):
-    today = date.today()
-    start_date = get_london_midnight_in_utc(today)
-    end_date = get_london_midnight_in_utc(today + timedelta(days=1))
+    today = datetime.utcnow().date()
+    start_date = get_midnight_in_utc(today)
+    end_date = get_midnight_in_utc(today + timedelta(days=1))
 
     subquery = db.session.query(
         Notification.notification_type,
@@ -602,20 +553,20 @@ def dao_find_services_with_high_failure_rates(start_date, end_date, threshold=10
     return query.all()
 
 
-def get_live_services_with_organisation():
+def get_live_services_with_organization():
     query = db.session.query(
         Service.id.label("service_id"),
         Service.name.label("service_name"),
-        Organisation.id.label("organisation_id"),
-        Organisation.name.label("organisation_name")
+        Organization.id.label("organization_id"),
+        Organization.name.label("organization_name")
     ).outerjoin(
-        Service.organisation
+        Service.organization
     ).filter(
         Service.count_as_live.is_(True),
         Service.active.is_(True),
         Service.restricted.is_(False)
     ).order_by(
-        Organisation.name,
+        Organization.name,
         Service.name
     )
 

app/dao/templates_dao.py

@@ -1,19 +1,11 @@
 import uuid
 from datetime import datetime
 
-from flask import current_app
 from sqlalchemy import asc, desc
 
 from app import db
 from app.dao.dao_utils import VersionOptions, autocommit, version_class
-from app.dao.users_dao import get_user_by_id
-from app.models import (
-    LETTER_TYPE,
-    SECOND_CLASS,
-    Template,
-    TemplateHistory,
-    TemplateRedacted,
-)
+from app.models import Template, TemplateHistory, TemplateRedacted
 
 
 @autocommit
@@ -46,38 +38,6 @@ def dao_update_template(template):
     db.session.add(template)
 
 
-@autocommit
-def dao_update_template_reply_to(template_id, reply_to):
-    Template.query.filter_by(id=template_id).update(
-        {"service_letter_contact_id": reply_to,
-         "updated_at": datetime.utcnow(),
-         "version": Template.version + 1,
-         }
-    )
-    template = Template.query.filter_by(id=template_id).one()
-
-    history = TemplateHistory(**
-                              {
-                                  "id": template.id,
-                                  "name": template.name,
-                                  "template_type": template.template_type,
-                                  "created_at": template.created_at,
-                                  "updated_at": template.updated_at,
-                                  "content": template.content,
-                                  "service_id": template.service_id,
-                                  "subject": template.subject,
-                                  "postage": template.postage,
-                                  "created_by_id": template.created_by_id,
-                                  "version": template.version,
-                                  "archived": template.archived,
-                                  "process_type": template.process_type,
-                                  "service_letter_contact_id": template.service_letter_contact_id,
-                                  "broadcast_data": template.broadcast_data,
-                              })
-    db.session.add(history)
-    return template
-
-
 @autocommit
 def dao_redact_template(template, user_id):
     template.template_redacted.redact_personalisation = True
@@ -133,28 +93,3 @@ def dao_get_template_versions(service_id, template_id):
     ).order_by(
         desc(TemplateHistory.version)
     ).all()
-
-
-def get_precompiled_letter_template(service_id):
-    template = Template.query.filter_by(
-        service_id=service_id,
-        template_type=LETTER_TYPE,
-        hidden=True
-    ).first()
-    if template is not None:
-        return template
-
-    template = Template(
-        name='Pre-compiled PDF',
-        created_by=get_user_by_id(current_app.config['NOTIFY_USER_ID']),
-        service_id=service_id,
-        template_type=LETTER_TYPE,
-        hidden=True,
-        subject='Pre-compiled PDF',
-        content='',
-        postage=SECOND_CLASS
-    )
-
-    dao_create_template(template)
-
-    return template

app/dao/uploads_dao.py

@@ -1,4 +1,5 @@
 from datetime import datetime
+from os import getenv
 
 from flask import current_app
 from sqlalchemy import String, and_, desc, func, literal, text
@@ -20,7 +21,7 @@ from app.utils import midnight_n_days_ago
 def _get_printing_day(created_at):
     return func.date_trunc(
         'day',
-        func.timezone('Europe/London', func.timezone('UTC', created_at)) + text(
+        func.timezone(getenv("TIMEZONE", "America/New_York"), func.timezone('UTC', created_at)) + text(
             # We add 6 hours 30 minutes to the local created_at time so that
             # any letters created after 5:30pm get shifted into the next day
             "interval '6 hours 30 minutes'"
@@ -36,7 +37,7 @@ def _get_printing_datetime(created_at):
 
 
 def _naive_gmt_to_utc(column):
-    return func.timezone('UTC', func.timezone('Europe/London', column))
+    return func.timezone('UTC', func.timezone(getenv("TIMEZONE", "America/New_York"), column))
 
 
 def dao_get_uploads_by_service_id(service_id, limit_days=None, page=1, page_size=50):
@@ -51,7 +52,6 @@ def dao_get_uploads_by_service_id(service_id, limit_days=None, page=1, page_size
         func.coalesce(
             Job.processing_started, Job.created_at
         ) >= today - func.coalesce(ServiceDataRetention.days_of_retention, 7),
-        Job.contact_list_id.is_(None),
     ]
     if limit_days is not None:
         jobs_query_filter.append(Job.created_at >= midnight_n_days_ago(limit_days))
@@ -128,23 +128,3 @@ def dao_get_uploads_by_service_id(service_id, limit_days=None, page=1, page_size
     ).order_by(
         desc("processing_started"), desc("created_at")
     ).paginate(page=page, per_page=page_size)
-
-
-def dao_get_uploaded_letters_by_print_date(service_id, letter_print_date, page=1, page_size=50):
-    return db.session.query(
-        Notification,
-    ).join(
-        Template, Notification.template_id == Template.id
-    ).filter(
-        Notification.service_id == service_id,
-        Notification.notification_type == LETTER_TYPE,
-        Notification.api_key_id.is_(None),
-        Notification.status != NOTIFICATION_CANCELLED,
-        Template.hidden.is_(True),
-        _get_printing_day(Notification.created_at) == letter_print_date.date(),
-    ).order_by(
-        desc(Notification.created_at)
-    ).paginate(
-        page=page,
-        per_page=page_size,
-    )

app/dao/users_dao.py

@@ -1,6 +1,6 @@
 import uuid
 from datetime import datetime, timedelta
-from random import SystemRandom
+from secrets import randbelow
 
 from sqlalchemy import func
 from sqlalchemy.orm import joinedload
@@ -19,16 +19,9 @@ def _remove_values_for_keys_if_present(dict, keys):
         dict.pop(key, None)
 
 
-def create_secret_code():
-    return ''.join(get_non_repeating_random_digits(5))
-
-
-def get_non_repeating_random_digits(length):
-    output = [None] * length
-    for index in range(length):
-        while output[index] in {None, output[index - 1]}:
-            output[index] = str(SystemRandom().randrange(10))
-    return output
+def create_secret_code(length=6):
+    random_number = randbelow(10 ** length)
+    return "{:0{length}d}".format(random_number, length=length)
 
 
 def save_user_attribute(usr, update_dict=None):
@@ -143,12 +136,12 @@ def get_user_and_accounts(user_id):
     return User.query.filter(
         User.id == user_id
     ).options(
-        # eagerly load the user's services and organisations, and also the service's org and vice versa
+        # eagerly load the user's services and organizations, and also the service's org and vice versa
         # (so we can see if the user knows about it)
         joinedload('services'),
-        joinedload('organisations'),
-        joinedload('organisations.services'),
-        joinedload('services.organisation'),
+        joinedload('organizations'),
+        joinedload('organizations.services'),
+        joinedload('services.organization'),
     ).one()
 
 
@@ -164,7 +157,7 @@ def dao_archive_user(user):
     for service_user in service_users:
         db.session.delete(service_user)
 
-    user.organisations = []
+    user.organizations = []
 
     user.auth_type = EMAIL_AUTH_TYPE
     user.email_address = get_archived_db_column_value(user.email_address)

app/delivery/send_to_providers.py

@@ -10,7 +10,7 @@ from notifications_utils.template import (
     SMSMessageTemplate,
 )
 
-from app import create_uuid, db, notification_provider_clients, statsd_client
+from app import create_uuid, db, notification_provider_clients
 from app.celery.research_mode_tasks import (
     send_email_response,
     send_sms_response,
@@ -28,7 +28,6 @@ from app.models import (
     EMAIL_TYPE,
     KEY_TYPE_TEST,
     NOTIFICATION_SENDING,
-    NOTIFICATION_SENT,
     NOTIFICATION_STATUS_TYPES_COMPLETED,
     NOTIFICATION_TECHNICAL_FAILURE,
     SMS_TYPE,
@@ -38,13 +37,16 @@ from app.serialised_models import SerialisedService, SerialisedTemplate
 
 def send_sms_to_provider(notification):
     service = SerialisedService.from_id(notification.service_id)
-
+    message_id = None
     if not service.active:
         technical_failure(notification=notification)
         return
 
     if notification.status == 'created':
         provider = provider_to_use(SMS_TYPE, notification.international)
+        if not provider:
+            technical_failure(notification=notification)
+            return
 
         template_model = SerialisedTemplate.from_id_and_service_id(
             template_id=notification.template_id, service_id=service.id, version=notification.template_version
@@ -56,11 +58,9 @@ def send_sms_to_provider(notification):
             prefix=service.name,
             show_prefix=service.prefix_sms,
         )
-        created_at = notification.created_at
-        key_type = notification.key_type
         if service.research_mode or notification.key_type == KEY_TYPE_TEST:
             update_notification_to_sending(notification, provider)
-            send_sms_response(provider.name, str(notification.id), notification.to)
+            send_sms_response(provider.name, str(notification.id))
 
         else:
             try:
@@ -78,7 +78,7 @@ def send_sms_to_provider(notification):
                     'international': notification.international,
                 }
                 db.session.close()  # no commit needed as no changes to objects have been made above
-                provider.send_sms(**send_sms_kwargs)
+                message_id = provider.send_sms(**send_sms_kwargs)
             except Exception as e:
                 notification.billable_units = template.fragment_count
                 dao_update_notification(notification)
@@ -87,18 +87,7 @@ def send_sms_to_provider(notification):
             else:
                 notification.billable_units = template.fragment_count
                 update_notification_to_sending(notification, provider)
-
-        delta_seconds = (datetime.utcnow() - created_at).total_seconds()
-        statsd_client.timing("sms.total-time", delta_seconds)
-
-        if key_type == KEY_TYPE_TEST:
-            statsd_client.timing("sms.test-key.total-time", delta_seconds)
-        else:
-            statsd_client.timing("sms.live-key.total-time", delta_seconds)
-            if service.high_volume:
-                statsd_client.timing("sms.live-key.high-volume.total-time", delta_seconds)
-            else:
-                statsd_client.timing("sms.live-key.not-high-volume.total-time", delta_seconds)
+    return message_id
 
 
 def send_email_to_provider(notification):
@@ -108,8 +97,7 @@ def send_email_to_provider(notification):
         technical_failure(notification=notification)
         return
     if notification.status == 'created':
-        provider = provider_to_use(EMAIL_TYPE)
-
+        provider = provider_to_use(EMAIL_TYPE, False)
         template_dict = SerialisedTemplate.from_id_and_service_id(
             template_id=notification.template_id, service_id=service.id, version=notification.template_version
         ).__dict__
@@ -124,8 +112,6 @@ def send_email_to_provider(notification):
             template_dict,
             values=notification.personalisation
         )
-        created_at = notification.created_at
-        key_type = notification.key_type
         if service.research_mode or notification.key_type == KEY_TYPE_TEST:
             notification.reference = str(create_uuid())
             update_notification_to_sending(notification, provider)
@@ -144,23 +130,14 @@ def send_email_to_provider(notification):
             )
             notification.reference = reference
             update_notification_to_sending(notification, provider)
-        delta_seconds = (datetime.utcnow() - created_at).total_seconds()
-
-        if key_type == KEY_TYPE_TEST:
-            statsd_client.timing("email.test-key.total-time", delta_seconds)
-        else:
-            statsd_client.timing("email.live-key.total-time", delta_seconds)
-            if service.high_volume:
-                statsd_client.timing("email.live-key.high-volume.total-time", delta_seconds)
-            else:
-                statsd_client.timing("email.live-key.not-high-volume.total-time", delta_seconds)
 
 
 def update_notification_to_sending(notification, provider):
     notification.sent_at = datetime.utcnow()
     notification.sent_by = provider.name
     if notification.status not in NOTIFICATION_STATUS_TYPES_COMPLETED:
-        notification.status = NOTIFICATION_SENT if notification.international else NOTIFICATION_SENDING
+        notification.status = NOTIFICATION_SENDING
+
     dao_update_notification(notification)
 
 
@@ -168,9 +145,11 @@ provider_cache = TTLCache(maxsize=8, ttl=10)
 
 
 @cached(cache=provider_cache)
-def provider_to_use(notification_type, international=False):
+def provider_to_use(notification_type, international=True):
     active_providers = [
-        p for p in get_provider_details_by_notification_type(notification_type, international) if p.active
+        p for p in get_provider_details_by_notification_type(
+            notification_type, international
+        ) if p.active
     ]
 
     if not active_providers:
@@ -180,11 +159,10 @@ def provider_to_use(notification_type, international=False):
         raise Exception("No active {} providers".format(notification_type))
 
     if len(active_providers) == 1:
-        weights = [100]
+        chosen_provider = active_providers[0]
     else:
         weights = [p.priority for p in active_providers]
-
-    chosen_provider = random.choices(active_providers, weights=weights)[0]
+        chosen_provider = random.choices(active_providers, weights=weights)[0]  # nosec B311 - not sec/crypto related
 
     return notification_provider_clients.get_client_by_name_and_type(chosen_provider.identifier, notification_type)
 

app/docs/__init__.py

@@ -0,0 +1,11 @@
+from os import path
+
+from flask import Blueprint, current_app, send_file
+
+docs = Blueprint('docs', __name__, url_prefix='/docs')
+
+
+@docs.route('/openapi.yml', methods=['GET'])
+def send_openapi():
+    openapi_schema = path.join(current_app.root_path, '../docs/openapi.yml')
+    return send_file(openapi_schema, mimetype='text/yaml'), 200

app/email_branding/email_branding_schema.py

@@ -1,7 +1,7 @@
 from app.models import BRANDING_TYPES
 
 post_create_email_branding_schema = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
+    "$schema": "http://json-schema.org/draft-07/schema#",
     "description": "POST schema for getting email_branding",
     "type": "object",
     "properties": {
@@ -15,7 +15,7 @@ post_create_email_branding_schema = {
 }
 
 post_update_email_branding_schema = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
+    "$schema": "http://json-schema.org/draft-07/schema#",
     "description": "POST schema for getting email_branding",
     "type": "object",
     "properties": {

app/events/rest.py

@@ -11,6 +11,6 @@ register_errors(events)
 @events.route('', methods=['POST'])
 def create_event():
     data = request.get_json()
-    event = event_schema.load(data).data
+    event = event_schema.load(data)
     dao_create_event(event)
-    return jsonify(data=event_schema.dump(event).data), 201
+    return jsonify(data=event_schema.dump(event)), 201

app/govuk_alerts/rest.py

@@ -1,31 +0,0 @@
-from flask import Blueprint, jsonify
-
-from app.dao.broadcast_message_dao import dao_get_all_broadcast_messages
-from app.errors import register_errors
-from app.utils import get_dt_string_or_none
-
-govuk_alerts_blueprint = Blueprint(
-    "govuk-alerts",
-    __name__,
-    url_prefix='/govuk-alerts',
-)
-
-register_errors(govuk_alerts_blueprint)
-
-
-@govuk_alerts_blueprint.route('')
-def get_broadcasts():
-    broadcasts = dao_get_all_broadcast_messages()
-    broadcasts_dict = {"alerts": [{
-        "id": broadcast.id,
-        "reference": broadcast.reference,
-        "channel": broadcast.channel,
-        "content": broadcast.content,
-        "areas": broadcast.areas,
-        "status": broadcast.status,
-        "starts_at": get_dt_string_or_none(broadcast.starts_at),
-        "finishes_at": get_dt_string_or_none(broadcast.finishes_at),
-        "approved_at": get_dt_string_or_none(broadcast.approved_at),
-        "cancelled_at": get_dt_string_or_none(broadcast.cancelled_at),
-    } for broadcast in broadcasts]}
-    return jsonify(broadcasts_dict), 200

app/inbound_sms/inbound_sms_schemas.py

@@ -1,5 +1,5 @@
 get_inbound_sms_for_service_schema = {
-    "$schema": "http://json-schema.org/draft-04/schema#",
+    "$schema": "http://json-schema.org/draft-07/schema#",
     "description": "schema for parameters allowed when searching for to field=",
     "type": "object",
     "properties": {

app/inbound_sms/rest.py

@@ -1,5 +1,4 @@
 from flask import Blueprint, jsonify, request
-from notifications_utils.recipients import try_validate_and_format_phone_number
 
 from app.dao.inbound_sms_dao import (
     dao_count_inbound_sms_for_service,
@@ -30,9 +29,10 @@ def post_inbound_sms_for_service(service_id):
     form = validate(request.get_json(), get_inbound_sms_for_service_schema)
     user_number = form.get('phone_number')
 
-    if user_number:
-        # we use this to normalise to an international phone number - but this may fail if it's an alphanumeric
-        user_number = try_validate_and_format_phone_number(user_number, international=True)
+    # TODO update this for US formatting
+    # if user_number:
+    #     # we use this to normalise to an international phone number - but this may fail if it's an alphanumeric
+    #     user_number = try_validate_and_format_phone_number(user_number, international=True)
 
     inbound_data_retention = fetch_service_data_retention_by_notification_type(service_id, 'sms')
     limit_days = inbound_data_retention.days_of_retention if inbound_data_retention else 7