The performance platform is going away soon. The only stat that we do not have in our database is the processing time. Let me clarify the only statistic we don't have in our database that we can query efficiently is the processing time. Any queries on notification_history are too inefficient to use on a web page.
Processing time = the total number of normal/team emails and text messages plus the number of messages that have gone from created to sending within 10 seconds per whole day. We can then easily calculate the percentage of messages that were marked as sending under 10 seconds.
When running the purge command I found about 4 users who could not be
deleted because their user id was still referenced in the services table
as they had created the service yet they were not a member of that
service anymore.
I have fixed this by checking that if they are not a member but created
the service then we also delete the service for them.
Note, I've followed the previous convention of no tests for this
function. I've run it locally and executed the code path so there should
be no major flaws in the code. There is a small chance I wasn't able to
exactly replicate the state that existed in preview on my local but
hopefully it was close enough to be accurate.
Text messages have placeholders in their body.
Emails have them in their subject line too.
Letters have them in their body, subject line and contact block.
We were only looking in the the body and subject when processing a job,
therefore the thing assembling the letter was not looking in all the
CSV columns it needed to, because it hadn’t been told about any
placeholders in the contact block.
Fixing this means always making sure we use the correct `Template`
instance for the type of template we’re dealing with. Which we were
already doing in a different part of the codebase. So it makes sense to
reuse that.
Turns out we fixed the same bug for email subjects over 3 years ago:
3ed97151ee
* it doesn't delete service email reply to or letter contacts, or
contact lists. I don't think the contact lists will ever be an issue
but it doesn't hurt to add it to the list of things to remove.
* it doesn't remove users from organisations before deleting the users
there may be more tables that link to Service that should be deleted,
but for now just add these ones that I could spot.
the nightly task won't be affected, it'll just trigger three times more
sub-tasks.
this doesn't need to be a two-part deploy because we only trigger this
overnight, so as long as the deploy completes in daytime we don't need
to worry about celery task signatures
also remove the `read().splitlines()` pattern from csv.reader usage,
as it's redundant. also add .vscode to gitignore to accomodate my
hipster IDE choices, and add some lines to pytest.ini so just running
`py.test` uses some useful flags
There was a situation where the SMS was sent, the request timed out we got a 503 so the notification was put on the retry queue. However, the provider got the notification and sent the message. This means we didn't set the billable_units on the notification and the service will not be billed.
This PR is to fix the notifications that we can accurrately update.
The association is based on the email address of the user that created the service, meaning we need to look at version 1 of the service_history.
Then use the existing methods to find the organisation by email address, then add the association.
Added a command to get more detailed information about the letters that
are listed in the files of zip files sent that are stored on S3. This takes
one or more file paths as arguments and creates a CSV file with a row for each
letter.
Updated the 'migrate-data-to-ft-billing' command to populate the new
postage column of ft_billing. This will be populated with the
postage of the notification for letters, or 'none' for email or sms. We
need to ensure there are no null values in postage so that the postage
column can become part of the primary key later.
Also updated the query to get the right rate letter rate now that we are
updating rates in the letter_rates table.
sets all letters in notification history (and notifications) to
"second", so that there's no null letters for billing etc. Commits in
ten day chunks (up to ~30k letters)
