freshen up files

.github/dependabot.yml

@@ -9,4 +9,3 @@ updates:
     directory: "/" # Location of package manifests
     schedule:
       interval: "daily"
-

.github/workflows/adr-accepted.yml

@@ -81,5 +81,3 @@ jobs:
           --title "Add ADR ${{ steps.next.outputs.number }} to the repo" \
           --body "This pull request was opened automatically because #${{ github.event.issue.number }} was closed after being marked as an approved ADR. It contains a markdown file capturing the ADR body at the time the issue was closed. Please verify that the markdown is correct before merging!" || true
         gh pr merge $BRANCH --auto --squash || true
-
-    

CONTRIBUTING.md

@@ -16,4 +16,4 @@ As part of a U.S. government agency, the General Services Administration (GSA)
 
 This project is in the public domain within the United States, and applicable copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
 
-All contributions to this project will be released under the CC0 dedication. By submitting a pull request or issue, you are agreeing to comply with this waiver of copyright interest.
+All contributions to this project will be released under the CC0 dedication. By submitting a pull request or issue, you are agreeing to comply with this waiver of copyright interest.

LICENSE.md

@@ -50,4 +50,4 @@ As stated in [CONTRIBUTING](CONTRIBUTING.md), all contributions to this project
 
 ## Trademark
 
-GSA owns, reserves, and retains all rights, title, and interest in all trademarks owned by GSA. The User does not acquire any right or interest of any kind in any GSA trademark because of their use of the Modified Version or any of the modifications made to the Work by GSA.
+GSA owns, reserves, and retains all rights, title, and interest in all trademarks owned by GSA. The User does not acquire any right or interest of any kind in any GSA trademark because of their use of the Modified Version or any of the modifications made to the Work by GSA.

docs/adrs/0002-how-to-handle-timezones.md

@@ -31,7 +31,7 @@ We've also identified the following areas as pieces of the application and
 service that could be impacted by any timezone changes:
 
 - Reports by day (or specific month/year)
-- Jobs running at a reasonable time 
+- Jobs running at a reasonable time
 - Job schedules (we want users to understand when things will happen)
 - Scheduling sending of messages
 - UI listing of time messages were sent

docs/adrs/0004-designing-pilot-content-visibility.md

@@ -79,5 +79,3 @@ Mockup of Option 1:
 
 Mockup of Option 2:
 ![Option 2](https://github.com/GSA/notifications-api/assets/6556888/ea7442e4-745a-49d2-a90c-156cc6129356)
-
-

docs/adrs/0007-adr-manage-total-record-retention-dynamically.md

@@ -36,4 +36,4 @@ _No response_
 
 ### Next Steps
 
-#463
+#463

docs/adrs/0008-adr-handle-paid-quotas-at-the-organization-level.md

@@ -27,11 +27,11 @@ At this time, services will pull directly from the organization's quota. We may
 
 ### Author
 
-@stvnrlly, @ccostino, @tdlowden 
+@stvnrlly, @ccostino, @tdlowden
 
 ### Stakeholders
 
-@amyashida 
+@amyashida
 
 ### Next Steps
 

docs/all.md

@@ -6,6 +6,7 @@
   - [Onboarding](#onboarding)
   - [Setting up the infrastructure](#setting-up-the-infrastructure)
 - [Using the logs](#using-the-logs)
+- [`git` hooks](#git-hooks)
 - [Testing](#testing)
   - [CI testing](#ci-testing)
   - [Manual testing](#manual-testing)
@@ -231,6 +232,14 @@ Staging: https://logs.fr.cloud.gov/app/discover#/view/73d7c820-596e-11ee-a43a-09
 
 Once in the view, you'll likely want to adjust the time range in the upper right of the page.
 
+# `git` hooks
+
+We're using [`pre-commit`](https://pre-commit.com/) to manage hooks in order to automate common tasks or easily-missed cleanup. It's installed as part of `make bootstrap` and is limited to this project's virtualenv.
+
+The configuration is stored in `.pre-commit-config.yaml`. In that config, there are links to the repos from which the hooks are pulled, so hop through there if you want a detailed description of what each one is doing.
+
+We do not maintain any hooks in this repository.
+
 # Testing
 
 ```
@@ -428,7 +437,7 @@ There is a Flask command to wipe user-created data (users, services, etc.).
 The command should stop itself if it's run in a production environment, but, you know, please don't run it
 in a production environment.
 
-Running locally: 
+Running locally:
 
 ```
 flask command purge_functional_test_data -u <functional tests user name prefix>
@@ -445,7 +454,7 @@ cf run-task notify-api "flask command purge_functional_test_data -u <functional
 
 For these, we're using Flask commands, which live in [`/app/commands.py`](../app/commands.py).
 
-This includes things that might be one-time operations! If we're running it on production, it should be a Flask 
+This includes things that might be one-time operations! If we're running it on production, it should be a Flask
 command Using a command allows the operation to be tested, both with `pytest` and with trial runs in staging.
 
 To see information about available commands, you can get a list with:
@@ -473,7 +482,7 @@ that both end up at `persist_notification`, which writes to the database, and `p
 which enqueues the sending.
 
 For CSV uploads, the CSV is first stored in S3 and queued as a `Job`. When the job runs, it iterates
-through the rows, running `process_job.save_sms` to send notifications through `persist_notification` and 
+through the rows, running `process_job.save_sms` to send notifications through `persist_notification` and
 `provider_tasks.deliver_sms`.
 
 # Writing public APIs
@@ -667,7 +676,7 @@ For tasks that should happen before other stuff, there's a priority queue. Platf
 can set templates to use this queue.
 
 Currently, this queue doesn't do anything special. If the normal queue is very busy, it's
-possible that this queue will be faster merely because it's shorter. By the same logic, a 
+possible that this queue will be faster merely because it's shorter. By the same logic, a
 busy priority queue is likely to be _slower_ than the normal queue
 
 ## Celery scheduled tasks
@@ -814,7 +823,7 @@ Assuming that you have followed all steps to set up localstack successfully (see
    - Go to settings and set the organization for your service to 'Broadcast services' (scroll down to platform admin)
    - Go to settings and set your service to 'live' (scroll down to platform admin)
 5. Run your app 'locally'.  I.e. run `make run-procfile` on this project and `make run-flask` on the admin project
-6. Sign in.  Verify you are running with localstack.  I.e., you do NOT receive a text message on sign in.  Instead, 
+6. Sign in.  Verify you are running with localstack.  I.e., you do NOT receive a text message on sign in.  Instead,
    you see your authentication code in green in the api logs
 7. Go to send messages and upload your csv file and send your 100000 messages
 

docs/localstack.md

@@ -16,7 +16,7 @@ How to Use Localstack in Your Development Work
 
 ### Add LOCALSTACK_ENDPOINT_URL to Your .env File (One-Time)
 
-* Find the value in the sample.env file  (# LOCALSTACK_ENDPOINT_URL=http://localhost:4566).  
+* Find the value in the sample.env file  (# LOCALSTACK_ENDPOINT_URL=http://localhost:4566).
 * Copy and uncomment it into your .env file
 
 ### Run with Localstack (Recurring)
@@ -41,4 +41,4 @@ You should be able to send an SMS message in the UI and observe it in the dashbo
 over a period of five minutes.  And you should not receive a text message.
 
 NOTE: You will still be prompted for a 2FA code when you log in.  To get the code, look in the notification-api
-logs for "AUTHENTICATION_CODE:".
+logs for "AUTHENTICATION_CODE:".

migrations/alembic.ini

@@ -42,4 +42,4 @@ formatter = generic
 
 [formatter_generic]
 format = %(levelname)-5.5s [%(name)s] %(message)s
-datefmt = %H:%M:%S
+datefmt = %H:%M:%S

migrations/versions/0025_notify_service_data.py

@@ -62,12 +62,12 @@ def upgrade():
     template_history_insert = """INSERT INTO templates_history (id, name, template_type, created_at,
                                                                 content, archived, service_id,
                                                                 subject, created_by_id, version)
-                                 VALUES (:template_id, :template_name, :template_type, :time_now, 
+                                 VALUES (:template_id, :template_name, :template_type, :time_now,
                                  :content, False, :service_id, :subject, :user_id, 1)
                               """
     template_insert = """INSERT INTO templates (id, name, template_type, created_at,
                                                 content, archived, service_id, subject, created_by_id, version)
-                                 VALUES (:template_id, :template_name, :template_type, :time_now, 
+                                 VALUES (:template_id, :template_name, :template_type, :time_now,
                                  :content, False, :service_id, :subject, :user_id, 1)
                               """
     email_verification_content = """Hi ((name)),\n\nTo complete your registration for GOV.UK Notify please click the link below\n\n((url))"""

migrations/versions/0095_migrate_existing_svc_perms.py

@@ -41,8 +41,8 @@ def upgrade():
     conn.execute(
         """
     INSERT INTO service_permissions (service_id, permission, created_at)
-    SELECT id, 'sms', '2017-05-26 17:30:00.000000' FROM services 
-    WHERE id NOT IN (SELECT service_id FROM service_permissions 
+    SELECT id, 'sms', '2017-05-26 17:30:00.000000' FROM services
+    WHERE id NOT IN (SELECT service_id FROM service_permissions
     WHERE service_id=id AND permission='sms')
     """
     )
@@ -50,8 +50,8 @@ def upgrade():
     conn.execute(
         """
         INSERT INTO service_permissions (service_id, permission, created_at)
-        SELECT id, 'email', '2017-05-26 17:30:00.000000' FROM services 
-        WHERE id NOT IN (SELECT service_id FROM service_permissions 
+        SELECT id, 'email', '2017-05-26 17:30:00.000000' FROM services
+        WHERE id NOT IN (SELECT service_id FROM service_permissions
         WHERE service_id=id AND permission='email')
         """
     )
@@ -59,7 +59,7 @@ def upgrade():
     conn.execute(
         """
     INSERT INTO service_permissions (service_id, permission, created_at)
-    SELECT id, 'letter', '2017-05-26 17:30:00.000000' FROM services 
+    SELECT id, 'letter', '2017-05-26 17:30:00.000000' FROM services
     WHERE can_send_letters AND id NOT IN (SELECT service_id FROM service_permissions
     WHERE service_id=id AND permission='letter')
     """
@@ -67,8 +67,8 @@ def upgrade():
     conn.execute(
         """
         INSERT INTO service_permissions (service_id, permission, created_at)
-        SELECT id, 'international_sms', '2017-05-26 17:30:00.000000' FROM services 
-        WHERE can_send_international_sms AND id NOT IN (SELECT service_id FROM service_permissions 
+        SELECT id, 'international_sms', '2017-05-26 17:30:00.000000' FROM services
+        WHERE can_send_international_sms AND id NOT IN (SELECT service_id FROM service_permissions
         WHERE service_id=id AND permission='international_sms')
         """
     )

migrations/versions/0139_migrate_sms_allowance_data.py

@@ -37,10 +37,10 @@ def upgrade():
         "time_now": datetime.utcnow(),
     }
     insert_row_if_not_exist = """
-        INSERT INTO annual_billing 
-        (id, service_id, financial_year_start, free_sms_fragment_limit, created_at, updated_at) 
-         SELECT uuid_in(md5(random()::text)::cstring), id, :current_year, :default_limit, :time_now, :time_now 
-         FROM services WHERE id NOT IN 
+        INSERT INTO annual_billing
+        (id, service_id, financial_year_start, free_sms_fragment_limit, created_at, updated_at)
+         SELECT uuid_in(md5(random()::text)::cstring), id, :current_year, :default_limit, :time_now, :time_now
+         FROM services WHERE id NOT IN
         (select service_id from annual_billing)
     """
     conn.execute(text(insert_row_if_not_exist), input_params)

migrations/versions/0156_set_temp_letter_contact.py

@@ -25,7 +25,7 @@ def upgrade():
     history_query = """
         UPDATE templates_history t_history set service_letter_contact_id = (
             SELECT service_letter_contact_id from templates
-            where service_letter_contact_id is not null 
+            where service_letter_contact_id is not null
             and id = t_history.id
             and version = t_history.version
         )

migrations/versions/0171_add_org_invite_template.py

@@ -22,13 +22,13 @@ def upgrade():
     template_insert = """
         INSERT INTO templates (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False,
         :notify_service_id, :subject, :user_id, 1, :process_type, false)
     """
     template_history_insert = """
         INSERT INTO templates_history (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False,
         :notify_service_id, :subject, :user_id, 1, :process_type, false)
     """
 

migrations/versions/0191_ft_billing_pkey.py

@@ -14,8 +14,8 @@ down_revision = "0189_ft_billing_data_type"
 def upgrade():
     op.get_bind()
     op.execute("ALTER TABLE ft_billing DROP CONSTRAINT ft_billing_pkey")
-    sql = """ALTER TABLE ft_billing ADD CONSTRAINT 
-    ft_billing_pkey PRIMARY KEY 
+    sql = """ALTER TABLE ft_billing ADD CONSTRAINT
+    ft_billing_pkey PRIMARY KEY
     (bst_date, template_id, service_id, rate_multiplier, provider, notification_type, international, rate)"""
     op.execute(sql)
 
@@ -23,7 +23,7 @@ def upgrade():
 def downgrade():
     op.get_bind()
     op.execute("ALTER TABLE ft_billing DROP CONSTRAINT ft_billing_pkey")
-    sql = """ALTER TABLE ft_billing ADD CONSTRAINT 
-    ft_billing_pkey PRIMARY KEY 
+    sql = """ALTER TABLE ft_billing ADD CONSTRAINT
+    ft_billing_pkey PRIMARY KEY
     (bst_date, template_id, service_id, rate_multiplier, provider, notification_type, international)"""
     op.execute(sql)

migrations/versions/0265_add_confirm_edit_templates.py

@@ -22,13 +22,13 @@ def upgrade():
     template_insert = """
         INSERT INTO templates (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
     """
     template_history_insert = """
         INSERT INTO templates_history (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
     """
 

migrations/versions/0285_default_org_branding.py

@@ -16,14 +16,14 @@ import sqlalchemy as sa
 
 def upgrade():
     op.execute(
-        """UPDATE organisation SET email_branding_id = email_branding.id 
+        """UPDATE organisation SET email_branding_id = email_branding.id
     FROM email_branding
     WHERE email_branding.domain in (SELECT domain FROM domain WHERE domain.organisation_id = organisation.id)
     """
     )
 
     op.execute(
-        """UPDATE organisation SET letter_branding_id = letter_branding.id 
+        """UPDATE organisation SET letter_branding_id = letter_branding.id
     FROM letter_branding
     WHERE letter_branding.domain in (SELECT domain FROM domain WHERE domain.organisation_id = organisation.id)
     """

migrations/versions/0294_add_verify_reply_to_.py

@@ -21,15 +21,15 @@ def upgrade():
     template_insert = """
         INSERT INTO templates (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
     """
     template_history_insert = """
         INSERT INTO templates_history (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
-  
+
     """
 
     email_template_content = "\n".join(

migrations/versions/0298_add_mou_signed_receipt.py

@@ -102,15 +102,15 @@ def upgrade():
     insert_t = """
         INSERT INTO templates (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, current_timestamp, 
+        VALUES (:template_id, :template_name, :template_type, current_timestamp,
         :content, False, :notify_service_id, :subject, :user_id, 1, :process_type, false)
     """
     insert_th = """
             INSERT INTO templates_history (id, name, template_type, created_at, content, archived, service_id, subject,
             created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, current_timestamp, 
+        VALUES (:template_id, :template_name, :template_type, current_timestamp,
         :content, False, :notify_service_id, :subject, :user_id, 1, :process_type, false)
-        
+
         """
 
     for template in templates:

migrations/versions/0303_populate_services_org_id.py

@@ -34,7 +34,7 @@ def upgrade():
             UPDATE services_history
                SET organisation_id = :organisation_id
             WHERE id = :service_id
-              AND version = (select max(version) from services_history sh2 where id = services_history.id); 
+              AND version = (select max(version) from services_history sh2 where id = services_history.id);
         """
         conn.execute(
             text(history_sql),

migrations/versions/0312_populate_returned_letters.py

@@ -16,11 +16,11 @@ def upgrade():
     conn = op.get_bind()
     sql = """
         select id, service_id, reference, updated_at
-        from notification_history 
+        from notification_history
         where notification_type = 'letter'
         and notification_status = 'returned-letter'"""
     insert_sql = """
-        insert into returned_letters(id, reported_at, service_id, notification_id, created_at, updated_at) 
+        insert into returned_letters(id, reported_at, service_id, notification_id, created_at, updated_at)
         values(uuid_in(md5(random()::text)::cstring), :updated_at, :service_id, :id, now(), null)
     """
 

migrations/versions/0320_optimise_notifications.py

@@ -28,13 +28,13 @@ def upgrade():
         # PLEASE NOTE: that if you create index on production you need to add concurrently to the create statement,
         # however we are unable to do that inside a transaction like this upgrade method
         ix_notifications_service_id_composite = """
-        CREATE INDEX ix_notifications_service_id_composite 
+        CREATE INDEX ix_notifications_service_id_composite
             on notifications (service_id, notification_type, notification_status, created_at)
         """
         op.execute(ix_notifications_service_id_composite)
 
         ix_notifications_notification_type_composite = """
-            CREATE INDEX ix_notifications_notification_type_composite 
+            CREATE INDEX ix_notifications_notification_type_composite
             on notifications (notification_type, notification_status, created_at)
         """
         op.execute(ix_notifications_notification_type_composite)

migrations/versions/0327_idx_notification_history.py

@@ -21,13 +21,13 @@ def upgrade():
         op.execute("DROP INDEX IF EXISTS ix_notification_history_service_id_created_at")
 
         index = """
-            CREATE INDEX IF NOT EXISTS ix_notification_history_service_id_composite 
+            CREATE INDEX IF NOT EXISTS ix_notification_history_service_id_composite
             on notification_history(service_id, key_type, notification_type, created_at)
             """
         op.execute(index)
 
         composite_index = """
-        CREATE INDEX IF NOT EXISTS ix_notifications_notification_type_composite 
+        CREATE INDEX IF NOT EXISTS ix_notifications_notification_type_composite
             on notifications(notification_type, notification_status, created_at)
         """
         op.execute(composite_index)
@@ -43,19 +43,19 @@ def downgrade():
     if environment not in ["live", "production"]:
         op.execute(
             """
-            CREATE INDEX IF NOT EXISTS ix_notifications_service_id_created_at 
+            CREATE INDEX IF NOT EXISTS ix_notifications_service_id_created_at
             ON notifications(service_id, date(created_at))
             """
         )
         op.execute(
             """
-            CREATE INDEX IF NOT EXISTS ix_notification_history_created_at 
+            CREATE INDEX IF NOT EXISTS ix_notification_history_created_at
             on notification_history(created_at)
             """
         )
         op.execute(
             """
-            CREATE INDEX IF NOT EXISTS ix_notification_history_service_id_created_at 
+            CREATE INDEX IF NOT EXISTS ix_notification_history_service_id_created_at
             on notification_history(created_at)
             """
         )
@@ -75,13 +75,13 @@ def downgrade():
         )
         op.execute(
             """
-            CREATE INDEX IF NOT EXISTS ix_notification_history_notification_status 
+            CREATE INDEX IF NOT EXISTS ix_notification_history_notification_status
             on notification_history (notification_status)
             """
         )
         op.execute(
             """
-            CREATE INDEX IF NOT EXISTS ix_notification_history_notification_type 
+            CREATE INDEX IF NOT EXISTS ix_notification_history_notification_type
             on notification_history (notification_type)
             """
         )

migrations/versions/0347_add_dvla_volumes_template.py

@@ -23,15 +23,15 @@ def upgrade():
     template_insert = """
         INSERT INTO templates (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
     """
     template_history_insert = """
         INSERT INTO templates_history (id, name, template_type, created_at, content, archived, service_id, subject,
         created_by_id, version, process_type, hidden)
-        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id, 
+        VALUES (:template_id, :template_name, :template_type, :time_now, :content, False, :notify_service_id,
         :subject, :user_id, 1, :process_type, false)
-    
+
     """
 
     email_template_content = "\n".join(

migrations/versions/0351_unique_key_annual_billing.py

@@ -17,7 +17,7 @@ environment = os.environ["NOTIFY_ENVIRONMENT"]
 
 def upgrade():
     index = """
-        CREATE UNIQUE INDEX CONCURRENTLY uix_service_id_financial_year_start 
+        CREATE UNIQUE INDEX CONCURRENTLY uix_service_id_financial_year_start
         ON annual_billing (service_id, financial_year_start)
     """
     constraint = """

migrations/versions/0378_add_org_names.py

@@ -26,7 +26,7 @@ def upgrade():
     op.execute("UPDATE services_history SET organisation_type = 'federal';")
 
     # remove uk values
-    service_delete = """DELETE FROM organisation_types WHERE name IN 
+    service_delete = """DELETE FROM organisation_types WHERE name IN
                     ('central','local','nhs','nhs_central','nhs_local','emergency_service','school_or_college','nhs_gp')
                 """
     op.execute(service_delete)
@@ -36,7 +36,7 @@ def upgrade():
 
 def downgrade():
     # ### commands auto generated by Alembic - please adjust! ###
-    service_insert = """INSERT INTO organisation_types VALUES 
+    service_insert = """INSERT INTO organisation_types VALUES
                     ('central','','250000')
                     ('local','f','25000')
                     ('nhs','','25000')

test_csv_files/multiple_letter.csv

@@ -8,4 +8,4 @@ name6, street6, town6, postcode6
 name7, street7, town7, postcode7
 name8, street8, town8, postcode8
 name9, street9, town9, postcode9
-name0, street0, town0, postcode0
+name0, street0, town0, postcode0