Increased Gunicorn threads to 10 on staging to test if there is a
performance increase when testing request per second. Increased to 10.
Gunicorn recommend 2 x cores + 1 however on PaaS the number of cores
is not consistent.
* unused variables
* variables in loops overshadowing imports
* excepts with no defined exc type (tried to avoid `except Exception` too)
* history mapper is still too complex
* default variables should never be mutable
(saves us having to write the stuff ourselves). Also adds a small
click plugin to do datetime parsing.
Sample output:
```
[leohemsted:~/dev/api]$ flask command create_provider_rates --help
Usage: flask command create_provider_rates [OPTIONS]
Backfill rates for a given provider
Options:
-p, --provider_name [mmg|firetext|ses]
[required]
-c, --cost FLOAT Cost (pence) per message including decimals
[required]
-d, --valid_from DATE [required]
--help Show this message and exit.
[leohemsted:~/dev/api]$ flask command create_provider_rates -p ses -c 1.234 -d invalid
Usage: flask command create_provider_rates [OPTIONS]
Error: Invalid value for "-d" / "--valid_from": Could not parse datetime string "invalid" formatted as %Y-%m-%dT%H:%M:%S
```
this reduces the amount of error messages we log (we'll no longer log
at error level when build-dvla-file-for-job retries while waiting for
the task to finish), and make sure we retry in those cases above - db
or s3 having temporary troubl
This causes an issue when it hits the max retry limit, and tries to
throw your exception to let you deal with it - at this point it was
moaning because we pass in a string
if it's not defined, and we're inside an exception block celery uses
that instead.
sleep for 10 seconds to try and make sure that all worker threads
(either web api or celery) have finished before we delete when we
delete the DB is unbound from the app, which can cause
"permission denied for relation" psycopg2 errors.
Removed the REST endpoint and the DAO that it uses as the endpoint is
no longer used by the Admin UI and the DAO is not reused anywhere
else.
- Removed REST endpoint
- Removed DAO which gets the stats
- Removed associated tests of both methods
Rationale:
Sometimes, when deploying, we've seen errors while stopping the old
apps: "(psycopg2.ProgrammingError) permission denied for relation notifications".
When you call cf stop, it may not be entirely synchronous. Under the
hood, cloudfoundry has to do a whole bunch of things when you stop an
app - it has its own internal db of what app states are, and also has
to remove it from any load balancers etc, and also it has to actually
stop the app. We're not sure if the `cf stop` command guarantees that
your process has already terminated by the time that the command
returns.
In our Makefile, we call `cf stop`, followed by `cf delete`. One
posisble theory is that the process is still running when `cf stop`
exits, and then `cf delete` unbinds that service from the database,
removing all of it's users' permissions.
This isn't confirmed, however, this commit removes the `cf stop`
command to see if it solves the issue. PaaS team confirmed that
it's redundant - `cf delete` will carry out the same tasks under
the hood.
