The behaviour of stacking the version decorators does not work as
expected.
What you would expect to happen is that each decorator causes a history
row to be written for its respective model object.
What actually happens is that the first decorator adds history records
to the database session, but then causes the database session to commit.
This means that subsequent uses of this decorator find a clean session,
and therefore no changes to copy to their respective history tables.
This commit changes the intended use of the decorator so that it is only
used once per function, and accepts multiple definitions of what to
record history for. This way it can record everything that needs to go
into the history before doing anything that would risk flushing the
session.
This is fiendishly difficult error to discover on your own.
It’s caused when, during the creation of a row in the database, you run
a query on the same table, or a table that joins to the table you’re
inserting into. What I think is happening is that the database is forced
to flush the session before running the query in order to maintain
consistency.
This means that the session is clean by the time the history stuff comes
to do its work, so there’s nothing for it to copy into the history
table, and it silently fails to record history.
Hopefully raising an exception will:
- prevent this from failing silently
- save whoever comes across this issue in the future a whole load of
time
Currently when someone creates a service we match them to an
organisation. Then if the organisation has a default branding set, their
service gets that branding.
However none of the organisations yet have a default branding set up.
This commit migrates the data about the relationship between an
organisation and its branding from being inferred from the `domain`
field on the branding, to being a proper database relationship.
The association is based on the email address of the user that created the service, meaning we need to look at version 1 of the service_history.
Then use the existing methods to find the organisation by email address, then add the association.
This celery task was not decorated with the cronitor decorator so never
checked in with cronitor.
Adding the decorator will ensure this task is monitored.
The requisite cronitor key is in the credentials repository already.
Signed-off-by: Toby Lorne <toby.lornewelch-richards@digital.cabinet-office.gov.uk>
Added a command to get more detailed information about the letters that
are listed in the files of zip files sent that are stored on S3. This takes
one or more file paths as arguments and creates a CSV file with a row for each
letter.
We need this in the admin app while we still have pages that:
- talk about the data sharing and financial agreement
- but aren’t within a service (so can’t look at the service’s
organisation)
This is a get, but it deliberately won’t work if you pass it an email
address, in order not to put personally identifying information in our
logs.
When we send a zip file of letters to DVLA we expect them to send back an acknowledgement of those files.
Previously they named the files like NOTIFY.20180202091254.ACK.TXT and the contents would contain the name of the zip file we sent with a date of when they got it.
They have updated this format to mirror the format of the zip file because there was an instance where they sent 2 files of the same name so the later overwrote the first.
Since the name matches our name, there is no need to get the file from S3 but just compare file names.
the create_nightly_notification_status task runs at 00:30am UK time,
however this means that in summer datetime.today() will return the
wrong date as the server (which runs on UTC) will run the task at
23:30 (populating the wrong row in the table).
fix this to use nice tz aware functions
* call variables unambiguous things like `start_time` or `bst_date` to
reduce risk of passing in the wrong thing
* simplify the count_dict object - remove nested dict and start_date
fields as superfluous
* use static datetime objects in tests rather than calculating them
each time
The pp client converts to UTC using the convert_utc_to_bst notify util.
This requires a datatime not a date, pass it a datetime, and add an
assertion in an existing test.
I didn't want to use the midnight conversion util in the test.
Signed-off-by: Toby Lorne <toby.lornewelch-richards@digital.cabinet-office.gov.uk>
Changed the query to get the performance platform stats from ft_notification_status. But the date used for the query needed to be a date, not datetime so the equality worked.
query args from GET requests are put into our logs, and we should avoid
personal data (eg phone numbers) in them. Remove this old GET now that
it's not used by the admin app anymore
The previous query was including all notifications regardless of notification_status. I don't think that's right, it shouldn't include things like technical-failure or validation-failed. Thoughts?
I also need to remove the query that's no longer being used.
We've seen the SQLAlchemy "could not acquire connection" error in
production during heavy traffic. Since we have more gunicorn eventlet
workers than we have DB connections available some workers need to
wait for a DB connection to become available before they can proceed
with the request. There's a timeout set on how long a worker would
wait and if that timeout is exceeded the above exception is raised.
Currently, we're using at most 1000 out of 5000 max DB connections,
40% peak CPU usage on the DB instance and an average of 60% CPU on
API instances during heavy load. The number of DB connections is
proportionally similar in preview and staging.
This slightly increases the number of max DB connections per API
instance. This should improve our utilization of API instances by
increasing the number of workers that can communicate with the DB
concurrently while staying well within the max DB connections limit.
