Related to: https://github.com/alphagov/notifications-aws/pull/905
Previously this would fail because the Docker image we use for CI
builds doesn't have an 'environment.sh' file; it uses preset env
vars instead. This makes the command to upgrade the DB optional -
if the env file is missing, the error should be self evident.
if you run `make help` or just `make` then you get a nice list of the
tasks that you can run (or at least, the ones with help text added to
them.
We were missing these for some of the key commands that a developer
would want to know about.
By adding help text to them, they will now show up in `make` or `make
help` and saves a developer needing to go either look in the README or
go look in the Makefile to figure out what commands are available.
Note, there is no particular convention around which commands have help
comments. I don't think we need to figure out this but at the least, the
ones which developers may want to run locally I think should show up.
This apps works with v11, but not with v13. Adding '|| true' in the
Makefile means the 'bootstrap' rule can be run multiple times, even
if the DB already exists.
This is more consistent with how we run all other tasks. Note that
the virtual env setup is not generally applicable, and developers
of this repo should follow the guidance in the README.
for reasons unknown, using a file descriptor no longer works on
concourse (or in that docker container generally). It might be a change
within cf-cli v7 vs v6.
Either way, it does't work, so use a temporary file. Clean up the
temporary file afterwards.
If the command fails, the temporary file will still stick around, so
I've added the file to the /tmp/ folder instead. it's full of secret
keys and things so if you do have a deployment error while running
locally, you should make sure you clean up the file (make cf-rollback
and make clean will both do this for you).
a bug in cf-cli v6 caused us to get rate limited, one solution to this
is to bump the version of cf-cli we're using to version 7. This has a
few syntax changes as the old v3 commands become mainline.
To upgrade locally, grab the latest version from brew.
```sh
brew install cloudfoundry/tap/cf-cli@7
```
for use when we don't want API to serve any traffic, but paas is still
running. It's a simple nginx_buildpack app that is pushed separately,
and then two makefile commands that toggle the routes (and also
stop/start the nginx app).
For all endpoints/methods it returns a 503, with the response body.
```
{
"status_code": 503,
"errors": [
{
"error": "PlannedMaintenanceError",
"message": "We’re performing some essential updates. Notify will be back shortly. Please check https://status.notifications.service.gov.uk/ for more details"
}
]
}
```
NB: If you hit `/` it'll still return 404 - as this is defined in the
paas-proxy instance on aws.
* remove gosuuser - this means we can upgrade the base image to
something more modern and not have to faff around with gpg
* remove unnecessary commands - some things need to exist in the
makefile to keep jenkins happy
* remove concept of building separately - pip install requirements.txt
in the dockerfile
- We are running statsd exporter as an app with a public route for
Prometheus to scrape
- This updates preview to send statsd metrics over the CF internal
networking to the statsd exporter
- Removes the sidecar statsd exporters too
Downloads a linux binary from Github and puts it in the scripts folder.
This should add it to the package CI uploads to CF during deploy.
Unfortunately, since out .cfignore is a symlink to .gitignore we
can't add the file to be ignored by git, so it shows up in untracked
if `cf-deploy` is run locally.
newer versions of cf api don't allow you to have multiple apps per
manifest file. So, instead of our current inheritance based model, move
to the newer doc-dl/antivirus/template-preview approved jinja based
model.
the new single manifest.yml.j2 file sets a bunch of variables based on
the CF_APP variable - things like NOTIFY_APP_NAME, default instances,
etc. Then the manifest is built up to define all of the app options
based on these defaults. Things default to sensible values, which can
vary based on environment.
When adding new environment variables, you'll need to add them to the
manifest file. If they're json encoded lists, you'll need to pass them
back to the `tojson` filter, or jinja2 will print them as python lists,
with single quotes around strings.
remove pip-accel - it's not been updated in two years, and pins our
version of pip to a version that is several breaking changes old.
make sure commands work if you're already in a venv - mostly by
checking for presence of $VIRTUAL_ENV, and ensuring we use the correct
pip to install packages. Also clean up the commands a bit.
you need to `pip install celery[sqs]` to get the additional
dependencies that celery needs to use SQS queues - there are two libs -
boto3 and pycurl.
pycurl is a bunch of python handles around curl, so needs to be
installed from source so it can link to your curl/ssl libs. On paas and
in docker this works fine (needed to add `libcurl4-openssl-dev` to the
docker container), but on macos it can't find openssl. We need to pass
a couple of flags in:
* set the environment variable PYCURL_SSL_LIBRARY=openssl
* pass in the global options `build_ext` and `-I{openssl_headers_path}`.
As shown here:
https://github.com/pycurl/pycurl/issues/530#issuecomment-395403253
Env var is no biggie, but using any install-option flags disables
wheels for the whole pip install run. (See
https://github.com/pypa/pip/issues/2677 and
https://github.com/pypa/pip/issues/4118 for more context on the
install-options flags). A whole bunch of our dependencies don't
install nicely from source (but do from wheel), so this commit installs
pycurl separately as an initial step, with the requisite flags, and
then installs the rest of the requirements as before.
I've updated the makefile and bootstrap.sh files to reflect this, but
if you run `pip install -r requirements.txt` from scratch you will run
into issues.
We don't want pyup.io upgrading sub-dependencies listed in the
requirements.txt file since it does it whenever a new version is
available regardless of what our application dependencies require.
The list of top-level dependencies is moved to requirements-app.txt,
which is used by `make freeze-requirements` to generate the full
list of requirements in requirements.txt.
This is based on alphagov/digitalmarketplace-api#615, so rationale
from that PR applies here.
We had a problem with unpinned packages on new deployments leading
to failed tests (e.g. alphagov/notifications-admin#2144) which is
why we're implementing this now.
After re-evaluating pipenv again, this still seems like the least
disruptive approach:
* pyup.io has experimental support for Pipfile, but doesn't respect
version ranges or updating hashes in the lock file
* CloudFoundry buildpack recognizes and supports Pipfiles out of the
box, but the support is relatively new. For example until recently
CF would install dev packages during deployment. It's also based on
generating a requirements file from the Pipfile, which doesn't
properly support pinning VCS dependencies (eg it doesn't set the
#egg= version, meaning pip will not upgrade the package if it's
already installed).
* pipenv has a strict dependency resolution algorithm, which doesn't
appear to be well documented and can cause some unexpected failures.
For example, pipenv doesn't seem to be able to install `awscli-cwlogs`
package at all, believing it to have a version conflict for `botocore`
(which it doesn't list as a direct dependency) while neither `pip` nor
`pip-tools` highlight any issues with it.
* While trying out `pipenv install` on our list of dependencies it would
regularly fail to install utils with a "Will try again." message.
While the installation succeeds after a retry, this doesn't inspire
confidence.
* The switch to Pipfile and pipenv-managed virtualenvs requires a series
of changes to `make` targets and scripts - replacing `pip install` with
`pipenv`, removing references to requirements files and prefixing
commands with `pipenv run`. While it's likely to simplify the overall
process of managing dependencies, it would require time to properly
implement across our applications and environments (Jenkins, PaaS,
docker containers, and dev machines).
Sometimes you just wanna run some tests directly using the `pytest`
command. But you’re in a new shell, and have forgotten to do
`source environment_test.sh`. The screen fills with red, and your day
just got a little bit worse.
This commit will stop this from ever happening again, by making the
setting environment variables part of running Pytest. It does this with
a plugin called pytest-env[1].
pytest.ini is the standard way of configuring pytest. Creating this file
where it didn’t exist before changes the behaviour of pytest, in that
it will now look for tests in the same directory as the file, rather
than defaulting to the `tests/` directory. So we also have to explicitly
configure pytest[2] to tell it that it should only look in this
directory. Otherwise it gets lost in the weeds of `node_modules`.
This also changes the way that `SQLALCHEMY_DATABASE_URI` is overriden to
the convention used by this plugin.
1. https://github.com/MobileDynasty/pytest-env
2. https://docs.pytest.org/en/latest/customize.html#confval-testpaths
fixup! Remove environment_test.sh
we saw an issue where the app started, then immediately crashed due to
a setup error. However, jenkins had already returned positively, and
the deploy continued.
cf-deploy should fail if the app doesn't start up.
We do this by looking through the cloudfoundry events, and aborting
if there are any `app.crash` events for the new GUID.
sleep for 10 seconds to try and make sure that all worker threads
(either web api or celery) have finished before we delete when we
delete the DB is unbound from the app, which can cause
"permission denied for relation" psycopg2 errors.
Rationale:
Sometimes, when deploying, we've seen errors while stopping the old
apps: "(psycopg2.ProgrammingError) permission denied for relation notifications".
When you call cf stop, it may not be entirely synchronous. Under the
hood, cloudfoundry has to do a whole bunch of things when you stop an
app - it has its own internal db of what app states are, and also has
to remove it from any load balancers etc, and also it has to actually
stop the app. We're not sure if the `cf stop` command guarantees that
your process has already terminated by the time that the command
returns.
In our Makefile, we call `cf stop`, followed by `cf delete`. One
posisble theory is that the process is still running when `cf stop`
exits, and then `cf delete` unbinds that service from the database,
removing all of it's users' permissions.
This isn't confirmed, however, this commit removes the `cf stop`
command to see if it solves the issue. PaaS team confirmed that
it's redundant - `cf delete` will carry out the same tasks under
the hood.
