When we cloned the repository and started making modifications, we
didn't initially keep tests in step. This commit tries to get us to a
clean test run by skipping tests that are failing and removing some
that we no longer expect to use (MMG, Firetext), with the intention that
we will come back in future and update or remove them as appropriate.
To find all tests skipped, search for `@pytest.mark.skip(reason="Needs
updating for TTS:`. There will be a brief description of the work that
needs to be done to get them passing, if known. Delete that line to make
them run in a standard test run (`make test`).
This makes a few changes to:
- Make local development consistent with our other apps. It's now
faster to start Celery locally since we don't try to build the
image each time - this is usually quick, but unnecessary.
- Add support for connecting to a local Redis instance. Note that
the previous suggestion of "REDIS = True" was incorrect as this
would be turned into the literal string "True".
I've also co-located and extended the recipes in the Makefile to
make them a bit more visible.
as a team we primarily develop locally. However, we've been experiencing
issues with pycurl, a subdependency of celery, that is notoriously
difficult to install on mac. On top of the existing issues, we're also
seeing it conflict with pyproj in bizarre ways (where the order of
imports between pyproj and pycurl result in different configurations of
dynamically linked C libraries being loaded.
You are encouraged to attempt to install pycurl locally, following these
instructions: https://github.com/alphagov/notifications-manuals/wiki/Getting-Started#pycurl
However, if you aren't having any luck, you can instead now run celery
in a docker container.
`make run-celery-with-docker`
This will build a container, install the dependencies, and run celery
(with the default of four concurrent workers).
It will pull aws variables from your aws configuration as boto would
normally, and it will attempt to connect to your local database with the
user `postgres`. If your local database is configured differently (for
example, with a different user, or on a different port), then you can
set the SQLALCHEMY_DATABASE_URI locally to override that.
This was added by mistake - the Concourse pipeline never did this
previously, and errors if we try (the necessary environment vars
aren't present, even if we wanted to).
Related to: https://github.com/alphagov/notifications-aws/pull/905
Previously this would fail because the Docker image we use for CI
builds doesn't have an 'environment.sh' file; it uses preset env
vars instead. This makes the command to upgrade the DB optional -
if the env file is missing, the error should be self evident.
if you run `make help` or just `make` then you get a nice list of the
tasks that you can run (or at least, the ones with help text added to
them.
We were missing these for some of the key commands that a developer
would want to know about.
By adding help text to them, they will now show up in `make` or `make
help` and saves a developer needing to go either look in the README or
go look in the Makefile to figure out what commands are available.
Note, there is no particular convention around which commands have help
comments. I don't think we need to figure out this but at the least, the
ones which developers may want to run locally I think should show up.
This apps works with v11, but not with v13. Adding '|| true' in the
Makefile means the 'bootstrap' rule can be run multiple times, even
if the DB already exists.
This is more consistent with how we run all other tasks. Note that
the virtual env setup is not generally applicable, and developers
of this repo should follow the guidance in the README.
for reasons unknown, using a file descriptor no longer works on
concourse (or in that docker container generally). It might be a change
within cf-cli v7 vs v6.
Either way, it does't work, so use a temporary file. Clean up the
temporary file afterwards.
If the command fails, the temporary file will still stick around, so
I've added the file to the /tmp/ folder instead. it's full of secret
keys and things so if you do have a deployment error while running
locally, you should make sure you clean up the file (make cf-rollback
and make clean will both do this for you).
a bug in cf-cli v6 caused us to get rate limited, one solution to this
is to bump the version of cf-cli we're using to version 7. This has a
few syntax changes as the old v3 commands become mainline.
To upgrade locally, grab the latest version from brew.
```sh
brew install cloudfoundry/tap/cf-cli@7
```
for use when we don't want API to serve any traffic, but paas is still
running. It's a simple nginx_buildpack app that is pushed separately,
and then two makefile commands that toggle the routes (and also
stop/start the nginx app).
For all endpoints/methods it returns a 503, with the response body.
```
{
"status_code": 503,
"errors": [
{
"error": "PlannedMaintenanceError",
"message": "We’re performing some essential updates. Notify will be back shortly. Please check https://status.notifications.service.gov.uk/ for more details"
}
]
}
```
NB: If you hit `/` it'll still return 404 - as this is defined in the
paas-proxy instance on aws.
* remove gosuuser - this means we can upgrade the base image to
something more modern and not have to faff around with gpg
* remove unnecessary commands - some things need to exist in the
makefile to keep jenkins happy
* remove concept of building separately - pip install requirements.txt
in the dockerfile
- We are running statsd exporter as an app with a public route for
Prometheus to scrape
- This updates preview to send statsd metrics over the CF internal
networking to the statsd exporter
- Removes the sidecar statsd exporters too
Downloads a linux binary from Github and puts it in the scripts folder.
This should add it to the package CI uploads to CF during deploy.
Unfortunately, since out .cfignore is a symlink to .gitignore we
can't add the file to be ignored by git, so it shows up in untracked
if `cf-deploy` is run locally.
newer versions of cf api don't allow you to have multiple apps per
manifest file. So, instead of our current inheritance based model, move
to the newer doc-dl/antivirus/template-preview approved jinja based
model.
the new single manifest.yml.j2 file sets a bunch of variables based on
the CF_APP variable - things like NOTIFY_APP_NAME, default instances,
etc. Then the manifest is built up to define all of the app options
based on these defaults. Things default to sensible values, which can
vary based on environment.
When adding new environment variables, you'll need to add them to the
manifest file. If they're json encoded lists, you'll need to pass them
back to the `tojson` filter, or jinja2 will print them as python lists,
with single quotes around strings.
remove pip-accel - it's not been updated in two years, and pins our
version of pip to a version that is several breaking changes old.
make sure commands work if you're already in a venv - mostly by
checking for presence of $VIRTUAL_ENV, and ensuring we use the correct
pip to install packages. Also clean up the commands a bit.
you need to `pip install celery[sqs]` to get the additional
dependencies that celery needs to use SQS queues - there are two libs -
boto3 and pycurl.
pycurl is a bunch of python handles around curl, so needs to be
installed from source so it can link to your curl/ssl libs. On paas and
in docker this works fine (needed to add `libcurl4-openssl-dev` to the
docker container), but on macos it can't find openssl. We need to pass
a couple of flags in:
* set the environment variable PYCURL_SSL_LIBRARY=openssl
* pass in the global options `build_ext` and `-I{openssl_headers_path}`.
As shown here:
https://github.com/pycurl/pycurl/issues/530#issuecomment-395403253
Env var is no biggie, but using any install-option flags disables
wheels for the whole pip install run. (See
https://github.com/pypa/pip/issues/2677 and
https://github.com/pypa/pip/issues/4118 for more context on the
install-options flags). A whole bunch of our dependencies don't
install nicely from source (but do from wheel), so this commit installs
pycurl separately as an initial step, with the requisite flags, and
then installs the rest of the requirements as before.
I've updated the makefile and bootstrap.sh files to reflect this, but
if you run `pip install -r requirements.txt` from scratch you will run
into issues.
