From 4ad3c73967ced0898a1af810446ae116d59038b6 Mon Sep 17 00:00:00 2001 From: "stvnrlly (as a bot)" Date: Tue, 19 Sep 2023 20:04:45 +0000 Subject: [PATCH 1/2] add ADR 0007: ADR: Manage total record retention dynamically --- ...nage-total-record-retention-dynamically.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 docs/adrs/0007-adr-manage-total-record-retention-dynamically.md diff --git a/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md b/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md new file mode 100644 index 000000000..abd2639c5 --- /dev/null +++ b/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md @@ -0,0 +1,39 @@ +# ADR: Manage total record retention dynamically + +Status: Accepted +Date: + +### Context + +Currently, we have a global maximum for messages sent per day. This is designed to limit the amount of PII that we're holding at any given time. That limit is a Redis counter that auto-increments with each message sent. + +However, since we've also implemented #318 to immediately scrub and archive successful messages, the Redis counter and the records stored will be out of sync. We can expect our number of records with PII to be much lower. + +To provide senders with more flexibility, our limit should be based on the number of records in the `notifications` table. + +### Decision + +We will check the total count of the `notifications` table inside of `check_service_over_total_message_limit()`. This is located in `/app/notifications/validators.py`. This may require a new DAO function to access the data. + +We also considered: + +- Adding more checks to validate whether the number of messages in a CSV would go over the limit, but this doesn't seem necessary (yet). +- Creating a job to cache the number of records in a Redis key. This would save some database calls while sending messages, but the performance tradeoff is probably not worth the loss of precision. + + +### Consequences + +- We will have more database calls while sending messages. +- It's possible for message-sending to get blocked during a batch until the database is cleaned. + +### Author + +@stvnrlly & @ccostino + +### Stakeholders + +_No response_ + +### Next Steps + +#463 \ No newline at end of file From 5a4e828e3f76f0fa4bf773e4d0323309fffc40a7 Mon Sep 17 00:00:00 2001 From: stvnrlly Date: Wed, 27 Sep 2023 09:47:29 -0400 Subject: [PATCH 2/2] add date --- ...07-adr-manage-total-record-retention-dynamically.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md b/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md index abd2639c5..f0c654974 100644 --- a/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md +++ b/docs/adrs/0007-adr-manage-total-record-retention-dynamically.md @@ -1,8 +1,8 @@ -# ADR: Manage total record retention dynamically - -Status: Accepted -Date: - +# ADR: Manage total record retention dynamically + +Status: Accepted +Date: 09/27/23 + ### Context Currently, we have a global maximum for messages sent per day. This is designed to limit the amount of PII that we're holding at any given time. That limit is a Redis counter that auto-increments with each message sent.