Commit Graph

8700 Commits

Author SHA1 Message Date
Rebecca Law
6f9edf0c07 Merge pull request #2650 from alphagov/delay-10-before-killing
Change how the bash script is started.
2019-10-31 16:56:33 +00:00
Rebecca Law
fe18512dd2 Change how the bash script is started.
By adding `exec` to the entrypoint bash script for the application, we can trap an EXIT from the script and execute our custom `on_exit` method with checks if the application process is busy before terminating, waiting up to 10 seconds. We don't need to trap `TERM` so that's been removed again.

Written by:
@servingupaces
@tlwr
2019-10-31 16:41:16 +00:00
Katie Smith
e504410ced 🎉 Pytest 5 🎉 2019-10-31 15:50:37 +00:00
Katie Smith
8abe427cb7 Fix tests which call str() on exception messages
Since Pytest 5, `ExceptionInfo` objects (returned by `pytest.raises`) now
have the same `str` representation as `repr`. This means that `str(e)`
now needs to be changed to `str(e.value)`.

https://github.com/pytest-dev/pytest/issues/5412
2019-10-31 15:38:44 +00:00
Katie Smith
04c1c35efb Stop calling fixures as functions in the tests 2019-10-31 15:04:23 +00:00
Katie Smith
97b807a99d Add test function to create service whitelist
This will be used to stop us needing to call the `sample_service_whitelist` fixture as
a function.
2019-10-31 15:04:23 +00:00
Katie Smith
87d7ed21a2 Fix fixtures
Stopped 2 fixtures from calling other fixtures as functions and causing
errors.
2019-10-31 15:04:23 +00:00
Rebecca Law
c35aca25e0 Merge pull request #2645 from alphagov/trap-term
Catch the TERM signal in the run_app_*paas scripts
2019-10-31 10:46:41 +00:00
Katie Smith
ac1642d8db Merge pull request #2646 from alphagov/pytest-5-changes
Start making tests Pytest 4 compliant
2019-10-30 13:56:47 +00:00
Katie Smith
38243cf860 Stop calling fixtures as functions in the tests 2019-10-30 13:05:53 +00:00
Katie Smith
fb875c50c1 Stop fixtures in conftest from calling other fixtures
Stopped some the fixtures in conftest.py from calling other
fixtures as functions to make them Pytest 4 compliant.
2019-10-30 11:22:36 +00:00
Katie Smith
99c3837dc1 Fix the fixtures for templates lacking certain permissions
These fixtures were both calling other fixtures as functions and being
called as functions in the tests. Rewriting the tests to make them
Pytest 4 compatible means we are no longer using
`sample_template_without_letter_permission`, so this has been deleted.
2019-10-30 11:22:36 +00:00
David McDonald
6252dcf916 Merge pull request #2643 from alphagov/set-postage
Set postage
2019-10-30 10:22:33 +00:00
Toby Lorne
4ad2e30e52 Catch the TERM signal in the run_app_*paas scripts
When Cloud Foundry applications are to be rescheduled from one cell to
another, or they are stopped, they are sent a SIGTERM signal and 10
seconds later, a SIGKILL signal.

Currently the scripts trap the POSIX defined EXIT handler, rather than
the signal directly.

In order for the signal to properly be propagated to celery, and the
celery workers, the script should call the on_exit function when
receiving a TERM signal.

Signed-off-by: Toby Lorne <toby.lornewelch-richards@digital.cabinet-office.gov.uk>
Co-authored-by: Becca <rebecca.law@digital.cabinet.office.gov.uk>
Co-authored-by: Toby <toby.lornewelch-richards@digital.cabinet.office.gov.uk>
2019-10-29 17:21:16 +00:00
Pea Tyczynska
c2825e10b1 Grab metadata when getting pdf letter preview from S3
Also use this metadata to decide whether preview pages need
overlay or not. So far we have always added overlay when validation
has failed. Now we will only show it when validation failed due to
content being outside of printable area.
2019-10-29 16:19:50 +00:00
Leo Hemsted
ca96c5d81b Merge pull request #2638 from alphagov/remove-loadtesting
remove loadtesting from providers
2019-10-29 11:30:09 +00:00
David McDonald
bc1b67fe4d Merge pull request #2644 from alphagov/makefile-help
Improve help comments for makefile
2019-10-29 10:23:53 +00:00
Katie Smith
1a4576f67e Merge pull request #2642 from alphagov/celery-task-id
Add request_ids to some Celery tasks
2019-10-28 13:34:58 +00:00
David McDonald
bf1ba9d9df Improve help comments for makefile
- Adds help text for freeze-requirements
- moves help text into correct position for upload-paas-artifact
2019-10-28 11:26:55 +00:00
Katie Smith
ceb7cee009 Pass request_id to tasks if available
We want to pass the `request_id` to Celery tasks if the task is called
from an HTTP request, so that we can add the `request_id` to the logs.
This change overwrites `apply_async` to add the `request_id` to the
kwargs if available. When we call the task, we then add the `request_id`
to g on Flask's application context.

Tasks called from `send_task` won't have a `request_id` for now, and
this change only affects tasks called from HTTP requests (not from other
tasks or from Celery Beat).
2019-10-28 10:59:25 +00:00
Katie Smith
1d6716275a Bump utils to bring in changes to logging
Freezing the requirements changes the Werkzeug version to 0.16.0, which
requires a change in how we import an exception
(https://github.com/pallets/werkzeug/blob/master/CHANGES.rst#version-0160)
2019-10-28 10:54:19 +00:00
Rebecca Law
98c61f58b1 Merge pull request #2624 from alphagov/add-logs-for-jobs
Add more logging for process_job
2019-10-28 09:53:15 +00:00
David McDonald
714c349efd Use existing postage format for validation
Also adds explicit test for checking of postage
2019-10-25 16:13:22 +01:00
David McDonald
d1195dfd5a Use json schema to validate post request to send PDF letter
This approach was suggested as the way to do validation things on
Notify.
2019-10-25 14:36:13 +01:00
David McDonald
7f708153d2 Require 'postage' key when sending a pdf letter 2019-10-24 16:12:47 +01:00
Rebecca Law
8794f770b9 Merge pull request #2640 from alphagov/remove-gunicorn-header
remove gunicorn server header from responses
2019-10-24 13:20:50 +01:00
Leo Hemsted
acb5e9fd47 remove gunicorn server header from responses
All our requests identify the web server they were served with via the
`server` response header. This opens us up to potential attackers who
might be crawling the internet looking for specific versions with known
vulnerabilities. As our dependencies are open source, this doesn't
affect any targeted attacks as they can just look at our repos on
github, but this theoretically will marginally improve security.

Regardless, the header isn't useful [1], we're not the first people to
want to get rid of it, and gunicorn are in the process of at least
amending it to remove the version information [2].

This shouldn't have any impact on us, though an empty string will be
passed through to debug information in event of a crash. That's fine
though, as we already know what version we're running.

[1] https://www.fastly.com/blog/headers-we-dont-want
[2] https://github.com/benoitc/gunicorn/issues/825
2019-10-23 16:38:34 +01:00
Katie Smith
67a73f2054 Merge pull request #2637 from alphagov/check-trial-mode-uploaded-letters
Check service is not in trial mode for uploaded letters
2019-10-23 15:45:51 +01:00
Leo Hemsted
e094dd4bfd remove loadtesting from providers
we don't use it since we wrote our own provider stubs for performance
tests.

this removes it from the api - it's still in the DB and will be
retrieved by queries, but is set to disabled on prod
2019-10-23 11:45:07 +01:00
Leo Hemsted
496b6f4737 Merge pull request #2627 from alphagov/letter-alert-v3
Letter alert
2019-10-22 13:26:35 +01:00
Katie Smith
9b8e0ed856 Check service is not in trial mode for uploaded letters
This adds an api check that the service is not in trial mode when
someone tries to send an uploaded letter.
2019-10-22 09:17:52 +01:00
Pea (Malgorzata Tyczynska)
35602b96fc Merge pull request #2631 from alphagov/add-validation-message-to-s3-as-metadata
Put pdf letter validation failed message in S3 metadata
2019-10-18 10:29:54 +01:00
Rebecca Law
ad14f96b8d A small change to make the code just a little bit clearer. 2019-10-17 15:17:43 +01:00
Pea Tyczynska
6ee7ac6cac Refactor and harmonise metadata for invalid letters with those sent from admin app 2019-10-16 14:11:50 +01:00
Leo Hemsted
0340927a6e Merge pull request #2634 from alphagov/startup-timeout
set startup timeout to 10 minutes
2019-10-15 14:18:48 +01:00
Leo Hemsted
46b0cce17b set startup timeout to 10 minutes
5 minutes isn't long enough to scale api at load, leading to failed deploys
2019-10-15 12:09:06 +01:00
Leo Hemsted
dc02060de2 Merge pull request #2632 from alphagov/scaling-fix
scale up api to 25 instances before deploy on production
2019-10-15 11:07:14 +01:00
Katie Smith
c9743d70b4 Merge pull request #2633 from alphagov/update-noti-history-query
Update insert_update_notification_history to take a query limit
2019-10-14 17:01:03 +01:00
Katie Smith
fb80a9c92e Update insert_update_notification_history to take a query limit
The nightly job to delete email notifications was failing because it was
timing out (`psycopg2.errors.QueryCanceled: canceling statement due to statement timeout`).

This adds a query limit to the query which inserts or updates
notification history so that it only updates a maximum of 10000 rows at
a time.
2019-10-14 16:51:46 +01:00
Leo Hemsted
9c2ded00c1 scale up api to 25 instances before deploy on production
deploys take up to five minutes, during which notify-paas-autoscaler
can't scale the app. We saw 502s due to a large volume of traffic
coming in during that time, and we couldn't react cos we were
deploying.

scale up to 25 instances, the autoscaler won't be able to downscale
until after the deploy has finished.
2019-10-14 16:12:35 +01:00
Leo Hemsted
b886127c80 Merge pull request #2630 from alphagov/clean-dockerfile
clean up docker and makefile
2019-10-14 11:37:40 +01:00
Pea Tyczynska
0b65e75fe9 Format metadata correctly and use MetadataDirective to put new metadata in S3 object 2019-10-14 11:11:32 +01:00
Pea Tyczynska
0a617379c4 Put pdf letter validation failed message in S3 metadata
So it can be used to tell the user why the letter has failed validation
2019-10-11 17:24:48 +01:00
Leo Hemsted
9e64dcbb2f clean up docker and makefile
* remove gosuuser - this means we can upgrade the base image to
  something more modern and not have to faff around with gpg
* remove unnecessary commands - some things need to exist in the
  makefile to keep jenkins happy
* remove concept of building separately - pip install requirements.txt
  in the dockerfile
2019-10-11 13:55:21 +01:00
Leo Hemsted
8285ef5f89 only check for dvla response files on mon/weds/fri
dvla don't process 2nd class files on tues and thurs
2019-10-08 18:16:45 +01:00
David McDonald
0152dcf254 Merge pull request #2626 from alphagov/drop-datetime-dimension-table
Delete dm_datetime database table
2019-10-08 14:37:26 +01:00
David McDonald
4bc71f36bf Delete dm_datetime database table
No longer needed
2019-10-08 14:02:43 +01:00
David McDonald
546ffe5094 Merge pull request #2625 from alphagov/drop-dm-dimension
Drop datetime dimension model
2019-10-08 10:33:11 +01:00
David McDonald
a0ace380b1 Drop datetime dimension model
We have a table for datetime dimension that we no longer use and
I believe can be dropped.

First step is to remove the model and release the change. The next
step will be to then add a database migration to drop the actual
table. I believe we need to do it in this order and it can't
be done as a single PR.
2019-10-08 10:15:39 +01:00
Leo Hemsted
6965478afc explicitly name days in celery unit tests
make it easier to understand what's happening
2019-10-07 08:58:09 +01:00