Increasing the Gunicorn workers to 10 actually made the API decrease
in performance from ~200 requests/sec with with 20 API instances to
150 requests per/sec. The base value is 5 so this is to test how the API
performs with a slight increase.
Increased Gunicorn threads to 10 on staging to test if there is a
performance increase when testing request per second. Increased to 10.
Gunicorn recommend 2 x cores + 1 however on PaaS the number of cores
is not consistent.
* unused variables
* variables in loops overshadowing imports
* excepts with no defined exc type (tried to avoid `except Exception` too)
* history mapper is still too complex
* default variables should never be mutable
(saves us having to write the stuff ourselves). Also adds a small
click plugin to do datetime parsing.
Sample output:
```
[leohemsted:~/dev/api]$ flask command create_provider_rates --help
Usage: flask command create_provider_rates [OPTIONS]
Backfill rates for a given provider
Options:
-p, --provider_name [mmg|firetext|ses]
[required]
-c, --cost FLOAT Cost (pence) per message including decimals
[required]
-d, --valid_from DATE [required]
--help Show this message and exit.
[leohemsted:~/dev/api]$ flask command create_provider_rates -p ses -c 1.234 -d invalid
Usage: flask command create_provider_rates [OPTIONS]
Error: Invalid value for "-d" / "--valid_from": Could not parse datetime string "invalid" formatted as %Y-%m-%dT%H:%M:%S
```
this reduces the amount of error messages we log (we'll no longer log
at error level when build-dvla-file-for-job retries while waiting for
the task to finish), and make sure we retry in those cases above - db
or s3 having temporary troubl
This causes an issue when it hits the max retry limit, and tries to
throw your exception to let you deal with it - at this point it was
moaning because we pass in a string
if it's not defined, and we're inside an exception block celery uses
that instead.
sleep for 10 seconds to try and make sure that all worker threads
(either web api or celery) have finished before we delete when we
delete the DB is unbound from the app, which can cause
"permission denied for relation" psycopg2 errors.
