run migrations if app is down

normally we check the app's status page to see if migrations need
running. However, if the _status endpoint doesn't respond with 200, we
don't necessarily want to abort the deploy - we may be trying to deploy
a code fix that fixes that status endpoint for example.

We don't know whether to run the migrations or not, so err on the side
of caution by re-running the migration. The migration itself might be
the fix that gets the app working after all.

had to do a little song and dance because sometimes the response won't
be populated before an exception is thrown

scripts/check_if_new_migration.py

@@ -15,13 +15,24 @@ def get_latest_db_migration_to_apply():
 
 def get_current_db_version():
     api_status_url = '{}/_status'.format(os.getenv('API_HOST_NAME'))
-    response = requests.get(api_status_url)
 
-    if response.status_code != 200:
-        sys.exit('Could not make a request to the API: {}'.format())
-
-    current_db_version = response.json()['db_version']
-    return current_db_version
+    try:
+        response = requests.get(api_status_url)
+        response.raise_for_status()
+        current_db_version = response.json()['db_version']
+        return current_db_version
+    except requests.exceptions.ConnectionError:
+        print(f'Could not make web request to {api_status_url}', file=sys.stderr)
+        return ''
+    except Exception:  # we expect these to be either either a http status code error, or a json decoding error
+        print(
+            f'Could not read status endpoint!\n\ncode {response.status_code}\nresponse "{response.text}"',
+            file=sys.stderr
+        )
+        # if we can't make a request to the API, the API is probably down. By returning a blank string (which won't
+        # match the filename of the latest migration), we force the migration to run, as the code change to fix the api
+        # might involve a migration file.
+        return ''
 
 
 def run():