The admin app validation did not catch this problem. But the API did.
This PR is a small fix to catch the erorr thrown by the notifications-utils/recipient validation methods and return a 400 status rather than a 500.
This only solves the issue of the user seeing "We are experiencing technical difficulties" rather than "invalid email address"
The bug can be replicated if you enter use quotes when entering the email address.
More work needs to be done so that the admin app does the same validation as the api so the user sees a nice form validtion error rather than a 400 after clicking send.
See: https://www.pivotaltracker.com/story/show/154472625
- Also convert the files info to upper() for comparison rather than lower
because original file names are in upper case. The unit tests contain examples of the returned lists.
Since preview and staging environments don't have a full DVLA
integration they're likely to contain letter notifications in
a 'sending' state. To avoid spamming Deskpro we skip the check
unless we're in a production or test environment.
We should receive a response file from DVLA by 4pm the next working
day (next Monday for letters created on Friday, Saturday or Sunday).
Response file triggers a task to update the letters status from
'sending' to either 'failed' or 'delivered', at which point there
should be no letter notifications in the 'sending' state for that day.
To catch any errors in the process (eg a missing response file from
DVLA) we add a scheduled task that checks letter notifications for
previous day (or Friday when run on Monday) and raises a Deskpro
ticket if it finds any in a 'sending' state. We're checking letter
notifications based on the `sent_at` date, which is set when the
letter PDF is sent to DVLA (so for letters created after 5:30pm it
will be the next day).
The task runs at 4:30pm, which should give the response file processing
task enough time to finish if the file was uploaded at 4pm.
this means if we end up with some notifications sending and others not,
due to problems with the ftp connectivity for example, we don't re-send
those that worked.
As a reminder, letter pdf notifications start as created and stay that
way until we have sent the zip file to DVLA, at which point they are
updated to sending
#
Previously, if the SMS recipient was None there would be a 500 error
with no message displayed to the user. We now check if the recipient is
None and raise a BadRequestError if this is the case.
PR #1550 added the rate_limit column to the Service table.
This PR removes the rate limits from the config and uses rate_limit from
the Service model instead. Rate limits are still separated into 'team',
'normal' and 'test', but these values are the same for a service.
Pivotal story https://www.pivotaltracker.com/story/show/153992529
The history was not being updated properly, we think this is because the declaritive attribute is not being set propery by the property.
When reply_to: None it will update the service_letter_contact_id, but not the service_letter_contact, we think when the history_meta is build the history class and checking if the value is updated it depends which attribute it is checking first.
In order to fix this issue, there is a new dao method to update the reply_to on the Template and insert a new Template history.
It seems selecting the service_letter_contact in the validation method was causing SQLAlchemy to persist the object. When the dao was called to save the object nothing was different so we didn't persist the history object.
It may be time to take another look at how we version. :(
By replacing user-provided services with manifest environment variables
we avoid the need to set the application environment variables from the
service data.
Most of the variable names already match the service JSON keys, but we
need to rename the ones that don't (eg MMG and Firetext `api_key`) this
is done in a separate credentials PR.
During database upgrades and database fail overs there has been errors
because the database connection stays open, when a query is run the
query fails and the connection is re-established. To avoid these errors
shorter timeouts have been used to keep the connections from getting
stale.
- SQLALCHEMY_POOL_TIMEOUT timeout idle connections after 30 secs
- Updated SQLALCHEMY_POOL_RECYCLE to recycle the connection every 5 mins
See guide on optimistic disconnect handling - using the pool recycle
as a way to manage this:
http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-optimistic
If is_letter_contact_blank then the user has set the letter contact block to be blank on purpose
ELSE IF is_letter_contact_blank is false THEN use the template default
IF template default is blank THEN the service_letter_contact is blank use the service default
Grouping the letters into a maximum number of files is necessary because
the SQS task needs to be under a certain size. We also compress the task
when sending.
When we’re doing user research we often:
- start the task by inviting the participant to a service on Notify
- have them use a prototype version of the admin app, hosted on a
different domain
Currently we can’t do both of these things together, because the invite
emails always send people to `notifications.service.gov.uk` (because
it’s the API that sends the emails, and the prototype admin app points
at the production API).
This commit changes the API to optionally allow an instance of the admin
app to specify which domain should be used when generating invite links.
add collate-letter-pdfs task (name pending). This retrieves a list of
letter pdf files (just the metadata, not the actual data) from s3, and
loops through them, calling the ftp task zip-and-send-letter-pdfs. It
groups them up by adding them to lists while counting the total
filesize, if it gets over a certain filesize (currently set to 500mb)
it breaks at that chunk, sends off that list of files to the ftp app,
and then starts building up a new list.
DVLA have a hard 2gb limit on how big the zip files we can send is -
however we're going to be limited by the amount of memory on the ftp
app well before we get around to handling 2gb of pdf data - so the
limit is 500mb for now. We'll adjust it after we see how ftp performs.
SQL Alchemy config changes were made to decrease the downtime of the
application. The last test only had 1 min of downtime in the upgrade
period i.e. 40 mins. Tested without the config changes to double
check the change had the desired effect. Adding back in so we can test
the changes under load and performance test outside of upgrade.
SQL Alchemy config changes were made to decrease the downtime of the
application. The last test only had 1 min of downtime in the upgrade
period i.e. 40 mins. Reverting the changes so that the same process
can be followed to ensure the changes had the desired effect.
