darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2025-12-11 15:52:21 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
ae29ae28e5bdcd2440830b3791f0a8e1c36dfe7d
notifications-api/migrations/versions/0178_add_filename.py

32 lines
1.4 KiB
Python
Raw Normal View History

As it turns out the s3ftp used to mount the s3 bucket to our ftp server puts the file on s3 more than once. So the SNS topic is triggered more than once. We need to deal with this, it's ok when updating a notification status from delivered to delivered. But the DailySortedLetter counts are being doubled. Adding the file_name to the table as a unique key to get around this issue. It will mean we have multiple rows for each billing_day, but that's ok we can aggregate that. This will also give us a way to see which file created which count.
2018-03-14 17:04:58 +00:00
"""
Revision ID: 0178_add_filename
Revises: 0177_add_virus_scan_statuses
Create Date: 2018-03-14 16:15:01.886998
"""
from alembic import op
import sqlalchemy as sa
revision = '0178_add_filename'
down_revision = '0177_add_virus_scan_statuses'
def upgrade():
Handle duplicate calls to update_letter_notifications_statuses As it turns out the SNS topic is trigger more that once when a file is placed in S3, this is caused by a bug in the s3ftp software used to mount the S3 bucket to the FTP server. S3ftp performs the create file operation more than once. This is resulting in duplicate counts of DailySortedLetter (the counts of how many letters marked as sorted or unsorted, which affect how much the provider will charge Notify for the letter). This PR adds a new column to DailySortedLetter called file_name. A new unique constraint on billing_day + file_name is added. Each time we write a row to the table the counts are over written rather than aggregated. I am aware that this PR is not backwards compatiable. However, since the code is typically triggered once a day around 13:00 then it is very unlikely an exception will occur during the deploy. Also a complete migration of the data based on all our response files on S3 will be performed soon, meaning the data will be corrected. Also if an exception does occur it is after the updates to notification status has already occurred.
2018-03-15 10:49:18 +00:00
# Deleting the data here is ok because a full migration from the files on s3 is coming.
op.execute("DELETE FROM daily_sorted_letter")
As it turns out the s3ftp used to mount the s3 bucket to our ftp server puts the file on s3 more than once. So the SNS topic is triggered more than once. We need to deal with this, it's ok when updating a notification status from delivered to delivered. But the DailySortedLetter counts are being doubled. Adding the file_name to the table as a unique key to get around this issue. It will mean we have multiple rows for each billing_day, but that's ok we can aggregate that. This will also give us a way to see which file created which count.
2018-03-14 17:04:58 +00:00
op.add_column('daily_sorted_letter', sa.Column('file_name', sa.String(), nullable=True))
op.create_index(op.f('ix_daily_sorted_letter_file_name'), 'daily_sorted_letter', ['file_name'], unique=False)
op.create_unique_constraint('uix_file_name_billing_day', 'daily_sorted_letter', ['file_name', 'billing_day'])
op.drop_index('ix_daily_sorted_letter_billing_day', table_name='daily_sorted_letter')
op.create_index(op.f('ix_daily_sorted_letter_billing_day'), 'daily_sorted_letter', ['billing_day'], unique=False)
def downgrade():
op.drop_index(op.f('ix_daily_sorted_letter_billing_day'), table_name='daily_sorted_letter')
op.create_index('ix_daily_sorted_letter_billing_day', 'daily_sorted_letter', ['billing_day'], unique=True)
op.drop_constraint('uix_file_name_billing_day', 'daily_sorted_letter', type_='unique')
op.drop_index(op.f('ix_daily_sorted_letter_file_name'), table_name='daily_sorted_letter')
op.drop_column('daily_sorted_letter', 'file_name')
Reference in New Issue Copy Permalink