Turns out our tests spent a lot of time recreating the app for each test
case, which is quite intense.
This commit makes the fixture sessions level, so the app is only created
once per test session, not once per test function.
This cuts down the time taken to run the test suite to about 50 seconds.
It also makes the tests more parallelizable. Before this change going
from 4 to 8 processes made the tests slower. Now it cuts them down from
about 50 seconds to about 35 seconds[1]. So this commit also lets Pytest
choose the best number of processes to run, since on my machine it
chooses 8, which is the fastest.
Overall this means the
1. With a 2.2GHz quad-core Intel Core i7 processor on a 2015 MacBook Pro
The list of top-level dependencies is moved to requirements-app.txt,
which is used by `make freeze-requirements` to generate the full
list of requirements in requirements.txt.
(See alphagov/notifications-api#1938 for details.)
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
If a PR is going to fail because tests aren’t passing then you:
- should know about it as quick as possible
- shouldn’t waste precious Jenkins CPU running subsequent tests
This commit adds the `-x` flag to pytest, which stops the test run as
soon as one failing test is discovered.
flake8-print is a flake8 plugin that checks for `print()` statements in
Python files.
This should save us having to manually spot these when reviewing pull
requests.
The `--enable=T` flag needs to be set until this bug is fixed:
https://github.com/JBKahn/flake8-print/issues/27
Back on my old 2013 Macbook Pro, 2 was the optimum number of cores for the fastest test runs. Which is why I chose `-n2` when I originally introduced this flag (see 9ced677ec7).
Now I have a faster Macbook (same as every other developer), which has more cores to take advantage of.
We also use 4 cores when running the API tests.
Cores | Time taken to run tests
--- | ---
1 | 28.31s
2 | 18.15s
3 | 14.33s
4 | 12.32s
5 | 12.98s
6 | 13.49s
8 | 15.37s
PEP8 was renamed to pycodestyle; this issue explains why:
PyCQA/pycodestyle#466
This commit changes our tests to use pycodestyle instead of pep8.
No changes to our code were required as a result.
We can make the tests run slightly faster by parallelizing them across
multiple CPU cores:
- from some casual testing locally, 2 cores gives the optimum speedup
- Travis container-based builds have 2 CPU cores available[1]
- the net gain is about 20%, or 2 seconds
- unfortunately we can’t do this on the API because each test is still
using the same instance of the database
1. https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
- no config overrides - now all set in environment
- use different files for staging and live too allow for differently named env variables
- updates to run_app and run_tests scripts to set correct environment (test/development) so correct config picked up
- use environment file on deployed environments to pick correct config
Similar to how PEP8 enforces Python style, there are tools for front end code:
- jshint[1] for Javascript
- sass-lint[2] for SASS
This commit adds the Gulp plugins for both, and sets up some sensible rules
(which can be iterated on).
It also adds a command to `./scripts/run_tests.sh`, so that any errors in the
front end code will fail the build before it has a chance to be deployed.
1. http://jshint.com/
2. https://www.npmjs.com/package/sass-lint
