f8dc3936fc
We have seen multiple issues in production where healthchecks have failed for our applications as responses have taken longer than 1 second (the default health check invocation timeout) to respond and this has marked the instance as unhealthy and restarted it. This restarting has dropped inflight requests and caused 502s for our users. We are not entirely sure why the healthchecks sometimes take longer than expected. One hypothesis is large amounts of traffic slowing response times of the apps, however we have also seen contradictory evidence where health checks can still fail even when apps are getting very low levels of traffic. There could also be an issue with the actual healthcheck process itself. Regardless of the cause, we think by changing the timeout to 10 seconds it might stop our apps being restarted when they are infact still healthy enough to serve requests to users. Further investigation will also be done by the PaaS team into the health check process itself to see if this throws any more light on the situation. 10 seconds was a fairly abritary choice that was significantly longer than 1 second.
notifications-admin
GOV.UK Notify admin application.
Features of this application
- Register and manage users
- Create and manage services
- Send batch emails and SMS by uploading a CSV
- Show history of notifications
First-time setup
Brew is a package manager for OSX. The following command installs brew:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Languages needed
brew install node
NPM is Node's package management tool. n is a tool for managing
different versions of Node. The following installs n and uses the long term support (LTS)
version of Node.
npm install -g n
n lts
npm rebuild node-sass
The app runs within a virtual environment. We use mkvirtualenv for easier working with venvs
pip install virtualenvwrapper
mkvirtualenv -p /usr/local/bin/python3 notifications-admin
Install dependencies and build the frontend assets:
workon notifications-admin
./scripts/bootstrap.sh
Rebuilding the frontend assets
If you want the front end assets to re-compile on changes, leave this running in a separate terminal from the app
npm run watch
Create a local environment.sh file containing the following:
echo "
export NOTIFY_ENVIRONMENT='development'
export FLASK_APP=application.py
export FLASK_DEBUG=1
export WERKZEUG_DEBUG_PIN=off
"> environment.sh
AWS credentials
Your aws credentials should be stored in a folder located at ~/.aws. Follow Amazon's instructions for storing them correctly
Running the application
workon notifications-admin
./scripts/run_app.sh
Then visit localhost:6012
Updating application dependencies
requirements.txt file is generated from the requirements-app.txt in order to pin
versions of all nested dependencies. If requirements-app.txt has been changed (or
we want to update the unpinned nested dependencies) requirements.txt should be
regenerated with
make freeze-requirements
requirements.txt should be committed alongside requirements-app.txt changes.
Working with static assets
When running locally static assets are served by Flask at http://localhost:6012/static/…
When running on preview, staging and production there’s a bit more to it:
