Merge pull request #1152 from GSA/debug_wash

More debugging for issue where user cannot send all messages

.github/pull_request_template.md

@@ -1,79 +1,22 @@
 <!--
-Please follow the instructions found in this pull request template so that we
-have all of the relevant details needed for our work.
-
-At the minimum, please be sure to fill in all sections found below and also do
-the following:
-
-- Provide an appropriate and descriptive title for the pull request
-- Link the pull request to its corresponding issue (must be done after creating
-  the pull request itself)
-- Assign yourself as the author
-- Attach the appropriate labels to it
-- Set it to be on the Notify.gov project board
-- Select one or more reviewers from the team or mark the pull request as a draft
-  depending on its current state
-   - If the pull request is a draft, please be sure to add reviewers once it is
-     ready for review and mark it ready for review
-
-For each section, please delete the instructions/sample text (that includes this
-text, though it is wrapped in an HTML comment just in case) and put in your own
-information.  Thank you!
+Not sure what you should include or write in a pull request?  Please read the
+[pull request documentation in our docs!](https://github.com/GSA/notifications-api/blob/main/docs/all.md#pull-requests)
 -->
 
-*A note to PR reviewers: it may be helpful to review our
-[code review documentation](https://github.com/GSA/notifications-api/blob/main/docs/all.md#code-reviews)
-to know what to keep in mind while reviewing pull requests.*
+*A note to PR reviewers: it may be helpful to review our [code review documentation](https://github.com/GSA/notifications-api/blob/main/docs/all.md#code-reviews) to know what to keep in mind while reviewing pull requests.*
 
 ## Description
 
-Please enter a clear description about your proposed changes and what the
-expected outcome(s) is/are from there.  If there are complex implementation
-details within the changes, this is a great place to explain those details using
-plain language.
-
-This should include:
-
-- Links to issues that this PR addresses
-- Screenshots or screen captures of any visible changes, especially for UI work
-- Dependency changes
-
-If there are any caveats, known issues, follow-up items, etc., make a quick note
-of them here as well, though more details are probably warranted in the issue
-itself in this case.
+Please enter a detailed description here.
 
 ## TODO (optional)
 
-If you're opening a draft PR, it might be helpful to list any outstanding work,
-especially if you're asking folks to take a look before it's ready for full
-review.  In this case, create a small checklist with the outstanding items:
-
-- [ ] TODO item 1
-- [ ] TODO item 2
-- [ ] TODO item ...
+* [ ] TODO item 1
+* [ ] TODO item 2
+* [ ] TODO item ...
 
 ## Security Considerations
 
-Please think about the security compliance aspect of your changes and what the
-potential impacts might be.
-
-**NOTE:  Please be mindful of sharing sensitive information here!  If you're not
-sure of what to write, please ask the team first before writing anything here.**
-
-Relevant details could include (and are not limited to) the following:
-
-- Handling secrets/credential management (or specifically calling out that there
-  is nothing to handle)
-- Any adjustments to the flow of data in and out the system, or even within it
-- Connecting or disconnecting any external services to the application
-- Handling of any sensitive information, such as PII
-- Handling of information within log statements or other application monitoring
-  services/hooks
-- The inclusion of a new external dependency or the removal of an existing one
-- ... (anything else relevant from a security compliance perspective)
-
-There are some cases where there are no security considerations to be had, e.g.,
-updating our documentation with publicly available information.  In those cases
-it is fine to simply put something like this:
-
-- None; this is a documentation update with publicly available information.
+* Consideration 1
+* Consideration 2
+* Consideration ...

.github/workflows/deploy-demo.yml

@@ -21,8 +21,8 @@ jobs:
       uses: tj-actions/changed-files@v44
       with:
         files: |
-          terraform/demo
-          terraform/shared
+          terraform/demo/**
+          terraform/shared/**
           .github/workflows/deploy-demo.yml
     - name: Terraform init
       if: steps.changed-terraform-files.outputs.any_changed == 'true'

.github/workflows/deploy-prod.yml

@@ -25,8 +25,8 @@ jobs:
       uses: tj-actions/changed-files@v44
       with:
         files: |
-          terraform/production
-          terraform/shared
+          terraform/production/**
+          terraform/shared/**
           .github/workflows/deploy-prod.yml
     - name: Terraform init
       if: steps.changed-terraform-files.outputs.any_changed == 'true'

.github/workflows/deploy.yml

@@ -26,8 +26,8 @@ jobs:
       uses: tj-actions/changed-files@v44
       with:
         files: |
-          terraform/staging
-          terraform/shared
+          terraform/staging/**
+          terraform/shared/**
           .github/workflows/deploy.yml
     - name: Terraform init
       if: steps.changed-terraform-files.outputs.any_changed == 'true'

README.md

@@ -493,6 +493,14 @@ instructions above for more details.
   - [Celery scheduled tasks](./docs/all.md#celery-scheduled-tasks)
 - [Notify.gov](./docs/all.md#notifygov)
   - [System Description](./docs/all.md#system-description)
+- [Pull Requests](.docs/all.md#pull-requests)
+  - [Getting Started](.docs/all.md#getting-started)
+  - [Description](.docs/all.md#description)
+  - [TODO (optional)](.docs/all.md#todo-(optional))
+  - [Security Considerations](.docs/all.md#security-considerations)
+- [Code Reviews](.docs/all.md#code-reviews)
+  - [For the reviewer](.docs/all.md#for-the-reviewer)
+  - [For the author](.docs/all.md#for-the-author)
 - [Run Book](./docs/all.md#run-book)
   - [ Alerts, Notifications, Monitoring](./docs/all.md#-alerts-notifications-monitoring)
   - [ Restaging Apps](./docs/all.md#-restaging-apps)

app/commands.py

@@ -589,6 +589,17 @@ def process_row_from_job(job_id, job_row_number):
             )
 
 
+@notify_command(name="download-csv-file-by-name")
+@click.option("-f", "--csv_filename", required=True, help="csv file name")
+def download_csv_file_by_name(csv_filename):
+
+    bucket_name = (current_app.config["CSV_UPLOAD_BUCKET"]["bucket"],)
+    access_key = (current_app.config["CSV_UPLOAD_BUCKET"]["access_key_id"],)
+    secret = (current_app.config["CSV_UPLOAD_BUCKET"]["secret_access_key"],)
+    region = (current_app.config["CSV_UPLOAD_BUCKET"]["region"],)
+    print(s3.get_s3_file(bucket_name, csv_filename, access_key, secret, region))
+
+
 @notify_command(name="populate-annual-billing-with-the-previous-years-allowance")
 @click.option(
     "-y",
@@ -854,10 +865,11 @@ def promote_user_to_platform_admin(user_email_address):
 
 @notify_command(name="purge-csv-bucket")
 def purge_csv_bucket():
-    bucket_name = getenv("CSV_BUCKET_NAME")
-    access_key = getenv("CSV_AWS_ACCESS_KEY_ID")
-    secret = getenv("CSV_AWS_SECRET_ACCESS_KEY")
-    region = getenv("CSV_AWS_REGION")
+    bucket_name = (current_app.config["CSV_UPLOAD_BUCKET"]["bucket"],)
+    access_key = (current_app.config["CSV_UPLOAD_BUCKET"]["access_key_id"],)
+    secret = (current_app.config["CSV_UPLOAD_BUCKET"]["secret_access_key"],)
+    region = (current_app.config["CSV_UPLOAD_BUCKET"]["region"],)
+
     print("ABOUT TO RUN PURGE CSV BUCKET")
     s3.purge_bucket(bucket_name, access_key, secret, region)
     print("RAN PURGE CSV BUCKET")

docs/all.md

@@ -38,6 +38,11 @@
   - [Celery scheduled tasks](#celery-scheduled-tasks)
 - [Notify.gov](#notifygov)
   - [System Description](#system-description)
+- [Pull Requests](#pull-requests)
+  - [Getting Started](#getting-started)
+  - [Description](#description)
+  - [TODO (optional)](#todo-(optional))
+  - [Security Considerations](#security-considerations)
 - [Code Reviews](#code-reviews)
   - [For the reviewer](#for-the-reviewer)
   - [For the author](#for-the-author)
@@ -55,6 +60,9 @@
 - [Data Storage Policies \& Procedures](#data-storage-policies--procedures)
   - [Potential PII Locations](#potential-pii-locations)
   - [Data Retention Policy](#data-retention-policy)
+- [Debug messages not being sent](#debug-messages-not-being-sent)
+  - [Getting the file location and tracing what happens](#getting-the-file-location-and-tracing-what-happens)
+  - [Viewing the csv file](#viewing-the-csv-file)
 
 
 # Infrastructure overview
@@ -817,6 +825,97 @@ Notify.gov also provisions and uses two AWS services via a [supplemental service
 For further details of the system and how it connects to supporting services, see the [application boundary diagram](https://github.com/GSA/us-notify-compliance/blob/main/diagrams/rendered/apps/application.boundary.png)
 
 
+Pull Requests
+=============
+
+Changes are made to our applications via pull requests, which show a diff
+(the before and after state of all proposed changes in the code) of of the work
+done for that particular branch.  We use pull requests as the basis for working
+on Notify.gov and modifying the application over time for improvements, bug
+fixes, new features, and more.
+
+There are several things that make for a good and complete pull request:
+
+* An appropriate and descriptive title
+* A detailed description of what's being changed, including any outstanding work
+  (TODOs)
+* A list of security considerations, which contains information about anything
+  we need to be mindful of from a security compliance perspective
+* The proper labels, assignee, code reviewer, and other project metadata set
+
+
+### Getting Started
+
+When you first open a pull request, start off by making sure the metadata for it
+is in place:
+
+* Provide an appropriate and descriptive title for the pull request
+* Link the pull request to its corresponding issue (must be done after creating
+  the pull request itself)
+* Assign yourself as the author
+* Attach the appropriate labels to it
+* Set it to be on the Notify.gov project board
+* Select one or more reviewers from the team or mark the pull request as a draft
+  depending on its current state
+   * If the pull request is a draft, please be sure to add reviewers once it is
+     ready for review and mark it ready for review
+
+### Description
+
+Please enter a clear description about your proposed changes and what the
+expected outcome(s) is/are from there.  If there are complex implementation
+details within the changes, this is a great place to explain those details using
+plain language.
+
+This should include:
+
+* Links to issues that this PR addresses (especially if more than one)
+* Screenshots or screen captures of any visible changes, especially for UI work
+* Dependency changes
+
+If there are any caveats, known issues, follow-up items, etc., make a quick note
+of them here as well, though more details are probably warranted in the issue
+itself in this case.
+
+### TODO (optional)
+
+If you're opening a draft PR, it might be helpful to list any outstanding work,
+especially if you're asking folks to take a look before it's ready for full
+review.  In this case, create a small checklist with the outstanding items:
+
+* [ ] TODO item 1
+* [ ] TODO item 2
+* [ ] TODO item ...
+
+### Security Considerations
+
+Please think about the security compliance aspect of your changes and what the
+potential impacts might be.
+
+**NOTE:  Please be mindful of sharing sensitive information here!  If you're not sure of what to write, please ask the team first before writing anything here.**
+
+Relevant details could include (and are not limited to) the following:
+
+* Handling secrets/credential management (or specifically calling out that there
+  is nothing to handle)
+* Any adjustments to the flow of data in and out the system, or even within it
+* Connecting or disconnecting any external services to the application
+* Handling of any sensitive information, such as PII
+* Handling of information within log statements or other application monitoring
+  services/hooks
+* The inclusion of a new external dependency or the removal of an existing one
+* ... (anything else relevant from a security compliance perspective)
+
+There are some cases where there are no security considerations to be had, e.g.,
+updating our documentation with publicly available information.  In those cases
+it is fine to simply put something like this:
+
+* None; this is a documentation update with publicly available information.
+
+This way it shows that we still gave this section consideration and that nothing
+happens to apply in this scenario.
+
+
 Code Reviews
 ============
 
@@ -856,19 +955,19 @@ behavior and lack of professionalism is not acceptable or tolerated.**
 When performing a code review, it is helpful to keep the following guidelines in
 mind:
 
-- Be on the lookout for any sensitive information and/or leaked credentials,
+* Be on the lookout for any sensitive information and/or leaked credentials,
   secrets, PII, etc.
-- Ask and call out things that aren't clear to you; it never hurts to double
+* Ask and call out things that aren't clear to you; it never hurts to double
   check your understanding of something!
-- Check that things are named descriptively and appropriately and call out
+* Check that things are named descriptively and appropriately and call out
   anything that is not.
-- Check that comments are present for complex areas when needed.
-- Make sure the pull request itself is properly prepared - it has a clear
+* Check that comments are present for complex areas when needed.
+* Make sure the pull request itself is properly prepared - it has a clear
   description, calls out security concerns, and has the necessary labels, flags,
   issue link, etc., set on it.
-- Do not be shy about using the suggested changes feature in GitHub pull request
+* Do not be shy about using the suggested changes feature in GitHub pull request
   comments; this can help save a lot of time!
-- Do not be shy about marking a review with the `Request Changes` status - yes,
+* Do not be shy about marking a review with the `Request Changes` status - yes,
   it looks big and red when it shows up, but this is completely fine and not to
   be taken as a personal mark against the author(s) of the pull request!
 
@@ -896,14 +995,14 @@ behavior and lack of professionalism is not acceptable or tolerated.**
 
 When going over a review, it may be helpful to keep these perspectives in mind:
 
-- Approach the review with an open mind, curiosity, and appreciation.
-- If anything the reviewer(s) mentions is unclear to you, please ask for
+* Approach the review with an open mind, curiosity, and appreciation.
+* If anything the reviewer(s) mentions is unclear to you, please ask for
   clarification and engage them in further dialogue!
-- If you disagree with a suggestion or request, please say so and engage in an
+* If you disagree with a suggestion or request, please say so and engage in an
   open and respecful dialogue to come to a mutual understanding of what the
   appropriate next step(S) should be - accept the change, reject the change,
   take a different path entirely, etc.
-- If there are no issues with any suggested edits or requested changes, make
+* If there are no issues with any suggested edits or requested changes, make
   the necessary adjustments and let the reviewer(s) know when the work is ready
   for review again.
 
@@ -1224,3 +1323,25 @@ Data Retention Policy
 Seven (7) days by default. Each service can be set with a custom policy via `ServiceDataRetention` by a Platform Admin. The `ServiceDataRetention` setting applies per-service and per-message type and controls both entries in the `notifications` table as well as `csv` contact files uploaded to s3
 
 Data cleanup is controlled by several tasks in the `nightly_tasks.py` file, kicked off by Celery Beat.
+
+
+# Debug messages not being sent
+
+
+## Getting the file location and tracing what happens
+
+
+Ask the user to provide the csv file name.  Either the csv file they uploaded, or the one that is autogenerated when they do a one-off send and is visible in the UI
+
+Starting with the admin logs, search for this file name.  When you find it, the log line should have the file name linked to the job_id and the csv file location.  Save both of these.
+
+In the api logs, search by job_id.  Either you will see evidence of the job failing and retrying over and over (in which case search for a stack trace using timestamp), or you will ultimately get to a log line that links the job_id to a message_id.  In this case, now search by message_id.  You should be able to find the actual result from AWS, either success or failure, with hopefully some helpful info.
+
+## Viewing the csv file
+
+If you need to view the questionable csv file, run the following command:
+
+
+```
+cf run-task notify-api "flask command download_csv_file_by_name -f <file location found in admin logs>"
+```

notifications_utils/recipients.py

@@ -602,6 +602,26 @@ def validate_us_phone_number(number):
         raise InvalidPhoneError(exc._msg) from exc
 
 
+def show_mangled_number_clues(number):
+
+    translator = {
+        "1": "X",
+        "2": "X",
+        "3": "X",
+        "4": "X",
+        "5": "X",
+        "6": "X",
+        "7": "X",
+        "8": "X",
+        "9": "X",
+        "0": "X",
+    }
+    for key in translator:
+        number = number.replace(key, translator[key])
+
+    return number
+
+
 def validate_phone_number(number, international=False):
     if (not international) or is_us_phone_number(number):
         return validate_us_phone_number(number)
@@ -619,7 +639,11 @@ def validate_phone_number(number, international=False):
     except NumberParseException as exc:
         if exc._msg == "Could not interpret numbers after plus-sign.":
             raise InvalidPhoneError("Not a valid country prefix") from exc
-        raise InvalidPhoneError(exc._msg) from exc
+        if not isinstance(number, str):
+            raise InvalidPhoneError(f"Number must be string, not type {type(number)}")
+        raise InvalidPhoneError(
+            f"Invalid phone number looks like {show_mangled_number_clues(number)} {exc._msg}"
+        )
 
 
 validate_and_format_phone_number = validate_phone_number

poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.
 
 [[package]]
 name = "aiohttp"
@@ -111,13 +111,13 @@ frozenlist = ">=1.1.0"
 
 [[package]]
 name = "alembic"
-version = "1.13.1"
+version = "1.13.2"
 description = "A database migration tool for SQLAlchemy."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "alembic-1.13.1-py3-none-any.whl", hash = "sha256:2edcc97bed0bd3272611ce3a98d98279e9c209e7186e43e75bbb1b2bdfdbcc43"},
-    {file = "alembic-1.13.1.tar.gz", hash = "sha256:4932c8558bf68f2ee92b9bbcb8218671c627064d5b08939437af6d77dc05e595"},
+    {file = "alembic-1.13.2-py3-none-any.whl", hash = "sha256:6b8733129a6224a9a711e17c99b08462dbf7cc9670ba8f2e2ae9af860ceb1953"},
+    {file = "alembic-1.13.2.tar.gz", hash = "sha256:1ff0ae32975f4fd96028c39ed9bb3c867fe3af956bd7bb37343b54c9fe7445ef"},
 ]
 
 [package.dependencies]
@@ -204,17 +204,17 @@ tests-no-zope = ["attrs[tests-mypy]", "cloudpickle", "hypothesis", "pympler", "p
 
 [[package]]
 name = "awscli"
-version = "1.33.15"
+version = "1.33.18"
 description = "Universal Command Line Environment for AWS."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "awscli-1.33.15-py3-none-any.whl", hash = "sha256:5a8d7e68a4cf68afc9d9ba4bef511526eb71027360f95a1080d39158bc930083"},
-    {file = "awscli-1.33.15.tar.gz", hash = "sha256:54a8089edb6756da46addcfcd56fdca21307a121216a81ef542e17b284cbe9c9"},
+    {file = "awscli-1.33.18-py3-none-any.whl", hash = "sha256:4065a0c9ee7bd2281e0b04616242693abbe17cd9d7be966abc7a850d5044226d"},
+    {file = "awscli-1.33.18.tar.gz", hash = "sha256:800cae2c020dae7e86877e2b53dee637c19acc62de8084bc67e3434ac174ca35"},
 ]
 
 [package.dependencies]
-botocore = "1.34.133"
+botocore = "1.34.136"
 colorama = ">=0.2.5,<0.4.7"
 docutils = ">=0.10,<0.17"
 PyYAML = ">=3.10,<6.1"
@@ -403,17 +403,17 @@ files = [
 
 [[package]]
 name = "boto3"
-version = "1.34.131"
+version = "1.34.136"
 description = "The AWS SDK for Python"
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "boto3-1.34.131-py3-none-any.whl", hash = "sha256:05e388cb937e82be70bfd7eb0c84cf8011ff35cf582a593873ac21675268683b"},
-    {file = "boto3-1.34.131.tar.gz", hash = "sha256:dab8f72a6c4e62b4fd70da09e08a6b2a65ea2115b27dd63737142005776ef216"},
+    {file = "boto3-1.34.136-py3-none-any.whl", hash = "sha256:d41037e2c680ab8d6c61a0a4ee6bf1fdd9e857f43996672830a95d62d6f6fa79"},
+    {file = "boto3-1.34.136.tar.gz", hash = "sha256:0314e6598f59ee0f34eb4e6d1a0f69fa65c146d2b88a6e837a527a9956ec2731"},
 ]
 
 [package.dependencies]
-botocore = ">=1.34.131,<1.35.0"
+botocore = ">=1.34.136,<1.35.0"
 jmespath = ">=0.7.1,<2.0.0"
 s3transfer = ">=0.10.0,<0.11.0"
 
@@ -422,13 +422,13 @@ crt = ["botocore[crt] (>=1.21.0,<2.0a0)"]
 
 [[package]]
 name = "botocore"
-version = "1.34.133"
+version = "1.34.136"
 description = "Low-level, data-driven core of boto 3."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "botocore-1.34.133-py3-none-any.whl", hash = "sha256:f269dad8e17432d2527b97ed9f1fd30ec8dc705f8b818957170d1af484680ef2"},
-    {file = "botocore-1.34.133.tar.gz", hash = "sha256:5ea609aa4831a6589e32eef052a359ad8d7311733b4d86a9d35dab4bd3ec80ff"},
+    {file = "botocore-1.34.136-py3-none-any.whl", hash = "sha256:c63fe9032091fb9e9477706a3ebfa4d0c109b807907051d892ed574f9b573e61"},
+    {file = "botocore-1.34.136.tar.gz", hash = "sha256:7f7135178692b39143c8f152a618d2a3b71065a317569a7102d2306d4946f42f"},
 ]
 
 [package.dependencies]
@@ -1261,13 +1261,13 @@ tests = ["coverage", "coveralls", "dill", "mock", "nose"]
 
 [[package]]
 name = "faker"
-version = "25.8.0"
+version = "26.0.0"
 description = "Faker is a Python package that generates fake data for you."
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "Faker-25.8.0-py3-none-any.whl", hash = "sha256:4c40b34a9c569018d4f9d6366d71a4da8a883d5ddf2b23197be5370f29b7e1b6"},
-    {file = "Faker-25.8.0.tar.gz", hash = "sha256:bdec5f2fb057d244ebef6e0ed318fea4dcbdf32c3a1a010766fc45f5d68fc68d"},
+    {file = "Faker-26.0.0-py3-none-any.whl", hash = "sha256:886ee28219be96949cd21ecc96c4c742ee1680e77f687b095202c8def1a08f06"},
+    {file = "Faker-26.0.0.tar.gz", hash = "sha256:0f60978314973de02c00474c2ae899785a42b2cf4f41b7987e93c132a2b8a4a9"},
 ]
 
 [package.dependencies]
@@ -1305,18 +1305,18 @@ typing = ["typing-extensions (>=4.8)"]
 
 [[package]]
 name = "flake8"
-version = "7.0.0"
+version = "7.1.0"
 description = "the modular source code checker: pep8 pyflakes and co"
 optional = false
 python-versions = ">=3.8.1"
 files = [
-    {file = "flake8-7.0.0-py2.py3-none-any.whl", hash = "sha256:a6dfbb75e03252917f2473ea9653f7cd799c3064e54d4c8140044c5c065f53c3"},
-    {file = "flake8-7.0.0.tar.gz", hash = "sha256:33f96621059e65eec474169085dc92bf26e7b2d47366b70be2f67ab80dc25132"},
+    {file = "flake8-7.1.0-py2.py3-none-any.whl", hash = "sha256:2e416edcc62471a64cea09353f4e7bdba32aeb079b6e360554c659a122b1bc6a"},
+    {file = "flake8-7.1.0.tar.gz", hash = "sha256:48a07b626b55236e0fb4784ee69a465fbf59d79eec1f5b4785c3d3bc57d17aa5"},
 ]
 
 [package.dependencies]
 mccabe = ">=0.7.0,<0.8.0"
-pycodestyle = ">=2.11.0,<2.12.0"
+pycodestyle = ">=2.12.0,<2.13.0"
 pyflakes = ">=3.2.0,<3.3.0"
 
 [[package]]
@@ -2098,13 +2098,9 @@ files = [
     {file = "lxml-5.2.2-cp36-cp36m-win_amd64.whl", hash = "sha256:edcfa83e03370032a489430215c1e7783128808fd3e2e0a3225deee278585196"},
     {file = "lxml-5.2.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:28bf95177400066596cdbcfc933312493799382879da504633d16cf60bba735b"},
     {file = "lxml-5.2.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3a745cc98d504d5bd2c19b10c79c61c7c3df9222629f1b6210c0368177589fb8"},
-    {file = "lxml-5.2.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1b590b39ef90c6b22ec0be925b211298e810b4856909c8ca60d27ffbca6c12e6"},
     {file = "lxml-5.2.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b336b0416828022bfd5a2e3083e7f5ba54b96242159f83c7e3eebaec752f1716"},
-    {file = "lxml-5.2.2-cp37-cp37m-manylinux_2_28_aarch64.whl", hash = "sha256:c2faf60c583af0d135e853c86ac2735ce178f0e338a3c7f9ae8f622fd2eb788c"},
     {file = "lxml-5.2.2-cp37-cp37m-manylinux_2_28_x86_64.whl", hash = "sha256:4bc6cb140a7a0ad1f7bc37e018d0ed690b7b6520ade518285dc3171f7a117905"},
-    {file = "lxml-5.2.2-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:7ff762670cada8e05b32bf1e4dc50b140790909caa8303cfddc4d702b71ea184"},
     {file = "lxml-5.2.2-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:57f0a0bbc9868e10ebe874e9f129d2917750adf008fe7b9c1598c0fbbfdde6a6"},
-    {file = "lxml-5.2.2-cp37-cp37m-musllinux_1_2_aarch64.whl", hash = "sha256:a6d2092797b388342c1bc932077ad232f914351932353e2e8706851c870bca1f"},
     {file = "lxml-5.2.2-cp37-cp37m-musllinux_1_2_x86_64.whl", hash = "sha256:60499fe961b21264e17a471ec296dcbf4365fbea611bf9e303ab69db7159ce61"},
     {file = "lxml-5.2.2-cp37-cp37m-win32.whl", hash = "sha256:d9b342c76003c6b9336a80efcc766748a333573abf9350f4094ee46b006ec18f"},
     {file = "lxml-5.2.2-cp37-cp37m-win_amd64.whl", hash = "sha256:b16db2770517b8799c79aa80f4053cd6f8b716f21f8aca962725a9565ce3ee40"},
@@ -2493,6 +2489,7 @@ files = [
     {file = "msgpack-1.0.8-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:5fbb160554e319f7b22ecf530a80a3ff496d38e8e07ae763b9e82fadfe96f273"},
     {file = "msgpack-1.0.8-cp39-cp39-win32.whl", hash = "sha256:f9af38a89b6a5c04b7d18c492c8ccf2aee7048aff1ce8437c4683bb5a1df893d"},
     {file = "msgpack-1.0.8-cp39-cp39-win_amd64.whl", hash = "sha256:ed59dd52075f8fc91da6053b12e8c89e37aa043f8986efd89e61fae69dc1b011"},
+    {file = "msgpack-1.0.8-py3-none-any.whl", hash = "sha256:24f727df1e20b9876fa6e95f840a2a2651e34c0ad147676356f4bf5fbb0206ca"},
     {file = "msgpack-1.0.8.tar.gz", hash = "sha256:95c02b0e27e706e48d0e5426d1710ca78e0f0628d6e89d5b5a5b91a5f12274f3"},
 ]
 
@@ -3210,13 +3207,13 @@ files = [
 
 [[package]]
 name = "pycodestyle"
-version = "2.11.1"
+version = "2.12.0"
 description = "Python style guide checker"
 optional = false
 python-versions = ">=3.8"
 files = [
-    {file = "pycodestyle-2.11.1-py2.py3-none-any.whl", hash = "sha256:44fe31000b2d866f2e41841b18528a505fbd7fef9017b04eff4e2648a0fadc67"},
-    {file = "pycodestyle-2.11.1.tar.gz", hash = "sha256:41ba0e7afc9752dfb53ced5489e89f8186be00e599e712660695b7a75ff2663f"},
+    {file = "pycodestyle-2.12.0-py2.py3-none-any.whl", hash = "sha256:949a39f6b86c3e1515ba1787c2022131d165a8ad271b11370a8819aa070269e4"},
+    {file = "pycodestyle-2.12.0.tar.gz", hash = "sha256:442f950141b4f43df752dd303511ffded3a04c2b6fb7f65980574f0c31e6e79c"},
 ]
 
 [[package]]
@@ -3475,7 +3472,6 @@ files = [
     {file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
     {file = "PyYAML-6.0.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:855fb52b0dc35af121542a76b9a84f8d1cd886ea97c84703eaa6d88e37a2ad28"},
     {file = "PyYAML-6.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40df9b996c2b73138957fe23a16a4f0ba614f4c0efce1e9406a184b6d07fa3a9"},
-    {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a08c6f0fe150303c1c6b71ebcd7213c2858041a7e01975da3a99aed1e7a378ef"},
     {file = "PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6c22bec3fbe2524cde73d7ada88f6566758a8f7227bfbf93a408a9d86bcc12a0"},
     {file = "PyYAML-6.0.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8d4e9c88387b0f5c7d5f281e55304de64cf7f9c0021a3525bd3b1c542da3b0e4"},
     {file = "PyYAML-6.0.1-cp312-cp312-win32.whl", hash = "sha256:d483d2cdf104e7c9fa60c544d92981f12ad66a457afae824d146093b8c294c54"},
@@ -3635,13 +3631,13 @@ full = ["numpy"]
 
 [[package]]
 name = "redis"
-version = "5.0.6"
+version = "5.0.7"
 description = "Python client for Redis database and key-value store"
 optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "redis-5.0.6-py3-none-any.whl", hash = "sha256:c0d6d990850c627bbf7be01c5c4cbaadf67b48593e913bb71c9819c30df37eee"},
-    {file = "redis-5.0.6.tar.gz", hash = "sha256:38473cd7c6389ad3e44a91f4c3eaf6bcb8a9f746007f29bf4fb20824ff0b2197"},
+    {file = "redis-5.0.7-py3-none-any.whl", hash = "sha256:0e479e24da960c690be5d9b96d21f7b918a98c0cf49af3b6fafaa0753f93a0db"},
+    {file = "redis-5.0.7.tar.gz", hash = "sha256:8f611490b93c8109b50adc317b31bfd84fff31def3475b92e7e80bf39f48175b"},
 ]
 
 [package.extras]
@@ -3989,13 +3985,13 @@ pyasn1 = ">=0.1.3"
 
 [[package]]
 name = "s3transfer"
-version = "0.10.1"
+version = "0.10.2"
 description = "An Amazon S3 Transfer Manager"
 optional = false
-python-versions = ">= 3.8"
+python-versions = ">=3.8"
 files = [
-    {file = "s3transfer-0.10.1-py3-none-any.whl", hash = "sha256:ceb252b11bcf87080fb7850a224fb6e05c8a776bab8f2b64b7f25b969464839d"},
-    {file = "s3transfer-0.10.1.tar.gz", hash = "sha256:5683916b4c724f799e600f41dd9e10a9ff19871bf87623cc8f491cb4f5fa0a19"},
+    {file = "s3transfer-0.10.2-py3-none-any.whl", hash = "sha256:eca1c20de70a39daee580aef4986996620f365c4e0fda6a86100231d62f1bf69"},
+    {file = "s3transfer-0.10.2.tar.gz", hash = "sha256:0711534e9356d3cc692fdde846b4a1e4b0cb6519971860796e6bc4c7aea00ef6"},
 ]
 
 [package.dependencies]
@@ -4752,4 +4748,4 @@ multidict = ">=4.0"
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.12.2"
-content-hash = "b1b4bfbfdc1f5cc9ae9d090f35b235a62e9dbabc683a5b5a1d0d414605219b48"
+content-hash = "d3ca67b44f40fb25b724b8468e07d30901ddced875ffe5d6b6710a17e492b072"

pyproject.toml

@@ -8,11 +8,11 @@ readme = "README.md"
 
 [tool.poetry.dependencies]
 python = "^3.12.2"
-alembic = "==1.13.1"
+alembic = "==1.13.2"
 amqp = "==5.2.0"
 beautifulsoup4 = "==4.12.3"
-boto3 = "^1.34.131"
-botocore = "^1.34.133"
+boto3 = "^1.34.136"
+botocore = "^1.34.136"
 cachetools = "==5.3.3"
 celery = {version = "==5.4.0", extras = ["redis"]}
 certifi = ">=2022.12.7"
@@ -48,7 +48,7 @@ pyjwt = "==2.8.0"
 python-dotenv = "==1.0.1"
 sqlalchemy = "==2.0.31"
 werkzeug = "^3.0.3"
-faker = "^25.8.0"
+faker = "^26.0.0"
 async-timeout = "^4.0.3"
 bleach = "^6.1.0"
 geojson = "^3.1.0"
@@ -70,13 +70,13 @@ markupsafe = "^2.1.5"
 pycparser = "^2.22"
 python-dateutil = "^2.9.0.post0"
 pyyaml = "^6.0.1"
-s3transfer = "^0.10.1"
+s3transfer = "^0.10.2"
 six = "^1.16.0"
 urllib3 = "^2.2.2"
 webencodings = "^0.5.1"
 itsdangerous = "^2.2.0"
 jinja2 = "^3.1.4"
-redis = "^5.0.6"
+redis = "^5.0.7"
 requests = "^2.32.3"
 
 
@@ -86,7 +86,7 @@ bandit = "*"
 black = "^24.3.0"
 cloudfoundry-client = "*"
 exceptiongroup = "==1.2.1"
-flake8 = "^7.0.0"
+flake8 = "^7.1.0"
 flake8-bugbear = "^24.1.17"
 freezegun = "^1.5.1"
 honcho = "*"

terraform/demo/main.tf

@@ -21,15 +21,6 @@ module "database" {
   rds_plan_name = "micro-psql"
 }
 
-module "redis" { # default v6.2; delete after v7.0 resource is bound
-  source = "github.com/GSA-TTS/terraform-cloudgov//redis?ref=v1.0.0"
-
-  cf_org_name     = local.cf_org_name
-  cf_space_name   = local.cf_space_name
-  name            = "${local.app_name}-redis-${local.env}"
-  redis_plan_name = "redis-dev"
-}
-
 module "redis-v70" {
   source = "github.com/GSA-TTS/terraform-cloudgov//redis?ref=v1.0.0"
 

terraform/production/main.tf

@@ -21,22 +21,13 @@ module "database" {
   rds_plan_name = "small-psql-redundant"
 }
 
-module "redis" { # default v6.2; delete after v7.0 resource is bound
-  source = "github.com/GSA-TTS/terraform-cloudgov//redis?ref=v1.0.0"
-
-  cf_org_name     = local.cf_org_name
-  cf_space_name   = local.cf_space_name
-  name            = "${local.app_name}-redis-${local.env}"
-  redis_plan_name = "redis-3node-large"
-}
-
 module "redis-v70" {
   source = "github.com/GSA-TTS/terraform-cloudgov//redis?ref=v1.0.0"
 
   cf_org_name     = local.cf_org_name
   cf_space_name   = local.cf_space_name
   name            = "${local.app_name}-redis-v70-${local.env}"
-  redis_plan_name = "redis-dev"
+  redis_plan_name = "redis-3node-large"
   json_params = jsonencode(
     {
       "engineVersion" : "7.0",

tests/app/service/send_notification/test_send_notification.py

@@ -79,7 +79,7 @@ def test_should_reject_bad_phone_numbers(notify_api, sample_template, mocker):
             assert json_resp["result"] == "error"
             assert len(json_resp["message"].keys()) == 1
             assert (
-                "Invalid phone number: The string supplied did not seem to be a phone number."
+                "Invalid phone number: Invalid phone number looks like invalid The string supplied did not seem to be a phone number."  # noqa
                 in json_resp["message"]["to"]
             )
             assert response.status_code == 400

tests/app/user/test_rest.py

@@ -226,7 +226,7 @@ def test_cannot_create_user_with_empty_strings(admin_request, notify_db_session)
     assert resp["message"] == {
         "email_address": ["Not a valid email address"],
         "mobile_number": [
-            "Invalid phone number: The string supplied did not seem to be a phone number."
+            "Invalid phone number: Invalid phone number looks like  The string supplied did not seem to be a phone number."  # noqa
         ],
         "name": ["Invalid name"],
     }
@@ -949,7 +949,7 @@ def test_cannot_update_user_with_mobile_number_as_empty_string(
         _expected_status=400,
     )
     assert resp["message"]["mobile_number"] == [
-        "Invalid phone number: The string supplied did not seem to be a phone number."
+        "Invalid phone number: Invalid phone number looks like  The string supplied did not seem to be a phone number."  # noqa
     ]
 
 

tests/notifications_utils/test_recipient_validation.py

@@ -9,6 +9,7 @@ from notifications_utils.recipients import (
     get_international_phone_info,
     international_phone_info,
     is_us_phone_number,
+    show_mangled_number_clues,
     try_validate_and_format_phone_number,
     validate_and_format_phone_number,
     validate_email_address,
@@ -324,6 +325,11 @@ def test_phone_number_rejects_invalid_international_values(phone_number, error_m
     assert error_message == str(e.value)
 
 
+def test_show_mangled_number_clues():
+    x = show_mangled_number_clues("848!!-202?-2020$$")
+    assert x == "XXX!!-XXX?-XXXX$$"
+
+
 @pytest.mark.parametrize("email_address", valid_email_addresses)
 def test_validate_email_address_accepts_valid(email_address):
     try: