This celery task was not decorated with the cronitor decorator so never
checked in with cronitor.
Adding the decorator will ensure this task is monitored.
The requisite cronitor key is in the credentials repository already.
Signed-off-by: Toby Lorne <toby.lornewelch-richards@digital.cabinet-office.gov.uk>
Added a command to get more detailed information about the letters that
are listed in the files of zip files sent that are stored on S3. This takes
one or more file paths as arguments and creates a CSV file with a row for each
letter.
We need this in the admin app while we still have pages that:
- talk about the data sharing and financial agreement
- but aren’t within a service (so can’t look at the service’s
organisation)
This is a get, but it deliberately won’t work if you pass it an email
address, in order not to put personally identifying information in our
logs.
When we send a zip file of letters to DVLA we expect them to send back an acknowledgement of those files.
Previously they named the files like NOTIFY.20180202091254.ACK.TXT and the contents would contain the name of the zip file we sent with a date of when they got it.
They have updated this format to mirror the format of the zip file because there was an instance where they sent 2 files of the same name so the later overwrote the first.
Since the name matches our name, there is no need to get the file from S3 but just compare file names.
the create_nightly_notification_status task runs at 00:30am UK time,
however this means that in summer datetime.today() will return the
wrong date as the server (which runs on UTC) will run the task at
23:30 (populating the wrong row in the table).
fix this to use nice tz aware functions
* call variables unambiguous things like `start_time` or `bst_date` to
reduce risk of passing in the wrong thing
* simplify the count_dict object - remove nested dict and start_date
fields as superfluous
* use static datetime objects in tests rather than calculating them
each time
The pp client converts to UTC using the convert_utc_to_bst notify util.
This requires a datatime not a date, pass it a datetime, and add an
assertion in an existing test.
I didn't want to use the midnight conversion util in the test.
Signed-off-by: Toby Lorne <toby.lornewelch-richards@digital.cabinet-office.gov.uk>
Changed the query to get the performance platform stats from ft_notification_status. But the date used for the query needed to be a date, not datetime so the equality worked.
query args from GET requests are put into our logs, and we should avoid
personal data (eg phone numbers) in them. Remove this old GET now that
it's not used by the admin app anymore
The previous query was including all notifications regardless of notification_status. I don't think that's right, it shouldn't include things like technical-failure or validation-failed. Thoughts?
I also need to remove the query that's no longer being used.
also, it should default to last 7 days, not last 6 days. also change
count_inbound_sms to have the days passed in, so that it's more
explicit at the endpoint that we only return 7 days regardless of your
service's data retention
We need to back fill the daily_sorted_count tables, so we need to iterate through all the files. No need to update the notification status. So this task has been separated out.
Sometimes we have to make a few services for what really is one
service, for example GOV.UK Pay and GOV.UK Pay Direct Debit. We also
have our own test services which aren’t included in the count of live
services. We currently count these as one service by not including them
in the beta partners spreadsheet.
This adds a column to mark such services as ‘not counted’, which can
later be used to exclude them from reporting.
When creating a service it should inherit it’s organisation’s branding,
if that organisation has branding.
This wasn’t working because we were referring to the ID of the branding
when making the association, not the branding itself.
Now that notifications-admin is always sending through
folder_permissions, the folder_permissions column of the invited_user
table can be made non-nullable. The migration also backfills the column
(to []) to account for existing null values.
This sets the folder permissions for a user when adding them to a
service. If a user is being added to a service after accepting an
invite, we need to account for the possibility that the folders we are
trying to add them to have been deleted before they accepted the invite.
Updated the add_user_to_service endpoint to only handle data in the
'new' format (`{"permissions": [...]}` instead of `[permission_1, permission_2]`)
since Admin has been updated to send data the new way.
This change means that we no longer need the Marshmallow Permission
schema, so it can be deleted.
Added a new JSONB column, folder_permissions, to the invited_users table
to store a list of folders that an invited user can see. This is
nullable for now, but will be changed to be non-nullable and
back-populated later.
if we partially retry a day, we would create new zip files, containing
different letters (if some were processed succesfully). We need these
files to have different filenames to earlier zip files so that we can
avoid overwriting log data in zips_sent.
Hashing the filename means that we'll only overwrite if it was the same
file containing the same content.
DVLA don't care about the naming conventions of zip files, other than
it must start with `NOTIFY.` and end with `.ZIP`. So lets format the
date in a more readable way, and separate it from the batch number
previously ftp would name the files itself by giving them a timestamp
when uploading. we ran into issues with tasks being picked up multiple
times and as such, uploading duplicate files. By naming the file before
creating the task, we can avoid this issue.
Files are now named `NOTIFY.YYYYMMDD######.ZIP` where the number is a
counter that increments with each task we've issued in that run of
collate-letter-pdfs-for-day
The data posted to the `add_user_to_service` endpoint is currently sent as a
list of permissions:
`[{'permission': MANAGE_SETTINGS}, {'permission': MANAGE_TEMPLATES}]`.
This endpoint is going to also be used for folder permissions, so the
data now needs to be nested:
`{'permissions': [{'permission': MANAGE_SETTINGS}, {'permission': MANAGE_TEMPLATES}]}`
This changes the add_user_to_service endpoint to accept data in either
format. Once admin is sending data in the new format, the code can be
simplified.
