Commit Graph

6294 Commits

Author SHA1 Message Date
Rebecca Law
d5d2b3d2a6 Update insert to use select_from - this allows the insert query to run as a single bulk insert and should be more efficient. 2019-05-02 13:46:15 +01:00
Rebecca Law
c9265aab68 Don't do anything if the query doesn't yield results. 2019-05-01 15:07:59 +01:00
Rebecca Law
0def0b7fd0 We want to staop inserting and updating NotificationHistory each time we insert/update Notification.
This PR adds a function to upsert (insert or update if exists) NotificationHistory all the rows from Notification that we are about to delete in the nightly task. This will happen just before the delete function. Since it is a upsert query the function can be called more than once.
This should allow us remove all the insert/updates to NotificationHistory.

However, there is a consern that this will double the length of time the tasks take. So do we do these upserts in a separate task or in the same one?
2019-05-01 14:26:11 +01:00
Leo Hemsted
0b0a7cc57c Merge pull request #2464 from alphagov/unpin-runtime
unpin patch version of python
2019-04-30 15:14:04 +01:00
Sakis
bc058e9cb5 Merge pull request #2473 from alphagov/ensure-correct-cf-target
Explicitly target the right space before deploying
2019-04-30 14:34:17 +01:00
Katie Smith
6de67411df Merge pull request #2485 from alphagov/pin-pytest-xdist
Make pyup ignore pytest-xdist
2019-04-29 14:30:55 +01:00
Katie Smith
2f497f5557 Merge pull request #2474 from alphagov/pyup-update-cffi-1.12.2-to-1.12.3
Update cffi to 1.12.3
2019-04-29 14:19:06 +01:00
Katie Smith
f34c183806 Merge pull request #2478 from alphagov/pyup-update-moto-1.3.7-to-1.3.8
Update moto to 1.3.8
2019-04-29 14:18:59 +01:00
Katie Smith
8c33fadd4d Make pyup ignore pytest-xdist
The next available version of pytest-xdist is 1.28.0, which requires
pytest>=4.4. We currently have Pytest pinned to version 3.

https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst#pytest-xdist-1280-2019-04-02
2019-04-29 14:15:43 +01:00
pyup-bot
a22e7a0679 Update cffi from 1.12.2 to 1.12.3 2019-04-29 14:05:47 +01:00
Katie Smith
b214ea22e1 Merge pull request #2479 from alphagov/pyup-update-pytest-mock-1.10.2-to-1.10.4
Update pytest-mock to 1.10.4
2019-04-29 13:49:42 +01:00
Katie Smith
e27657c161 Merge pull request #2470 from alphagov/bump-requirements
Bump requirements
2019-04-29 13:49:28 +01:00
Rebecca Law
37e580eed3 Update requirements 2019-04-29 09:46:16 +01:00
Rebecca Law
ee69e70b76 Bump requirements.
Minor weekly updated and a bump for a security alert in urllib3.
Sort list in test, to ensure it passes everytime.
2019-04-26 10:49:25 +01:00
Katie Smith
fdb3d7e899 Merge pull request #2482 from alphagov/bump-utils
Bump utils to bring in changes to RecipientCSV rows
2019-04-26 10:09:03 +01:00
Katie Smith
c02b7edb92 Bump utils to bring in changes to RecipientCSV rows
Bumped utils to version 31.2.5, which changes when the rows of a
RecipientCSV get created. Switched to using `.get_rows()` from
RecipientCSV (a generator) instead of the `.rows` property (which builds
a list of the rows in memory).
2019-04-25 10:58:19 +01:00
Pea (Malgorzata Tyczynska)
3c4133e543 Merge pull request #2481 from alphagov/fix_bug_preview_overlay
Fix bug in precompiled preview with overlay
2019-04-25 10:34:02 +01:00
Pea Tyczynska
454690a008 Fix bug in precompiled preview with overlay
The bug treated all pages of png preview as if they were all first
page, showing overlay for address bar
2019-04-24 18:29:01 +01:00
pyup-bot
64f1e528c8 Update pytest-mock from 1.10.2 to 1.10.4 2019-04-24 16:06:13 +01:00
pyup-bot
c13118a8e4 Update moto from 1.3.7 to 1.3.8 2019-04-24 16:06:09 +01:00
Rebecca Law
a348e6bbf2 Merge pull request #2472 from alphagov/fix-populate-go-live
Fix command to populate go live data
2019-04-24 16:04:36 +01:00
Athanasios Voutsadakis
bf4e75da40 Explicitly target the right space before deploying 2019-04-24 15:55:45 +01:00
Rebecca Law
059bd52ee9 Leave go live user empty if it's not populated in spreadsheet 2019-04-24 15:44:57 +01:00
Rebecca Law
acd47f44f0 We want to continue not break. 2019-04-24 15:37:40 +01:00
Alexey Bezhan
bc7d91daec Merge pull request #2468 from alphagov/local-statsd-exporter
Local statsd exporter
2019-04-24 15:27:55 +01:00
Rebecca Law
b1d4af2f4a Merge pull request #2467 from alphagov/populate-go-live-data
Populate go live user and date
2019-04-24 14:28:03 +01:00
Alexey Bezhan
528b095fcc Add statsd exporter metric mapping configuration file
`statsd_mapping.yml` sets all timers to default to a Prometheus
histogram metric type and renames all metrics to start with `notifications_`.
2019-04-24 13:50:13 +01:00
Alexey Bezhan
570cbc3eab Add statsd_exporter to app PaaS startup scripts
`statsd_exporter` is only started if `STATSD_HOST` is set to `localhost`.
2019-04-24 13:50:13 +01:00
Alexey Bezhan
ba2abc9127 Add a local_statsd configuration to PaaS manifest template
Running `statsd_exporter` alongside the app process allows us to get
StatsD metrics pushed by workers to Prometheus.

This requires adding a route to the worker instances and binding the
RE prometheus discovery service. So this approach won't work for API
and admin since they already have `gunicorn` bound to the `$PORT`.

Since we're not ready to switch all apps to Prometheus metrics at once
and we don't currently have a way to push statsd metrics to multiple
destination we're using a configuration setting in the manifest template
to switch individual workers in specific environments.

`local_statsd` contains a list of environments where the app should
use local `statsd_exporter` for pushing statsd metrics instead of
HostedGraphite.
2019-04-24 13:50:13 +01:00
Alexey Bezhan
0138eb0cae Make statsd host configurable with an env variable
Setting `STATSD_HOST` for an env variable allows us to switch to a
local statsd_exporter on a per-app basis.

This also changes `STATSD_ENABLED` to be on when `STATSD_HOST` is set,
avoiding the need to set it separately.
2019-04-24 13:50:13 +01:00
Alexey Bezhan
394a0244d0 Merge pull request #2469 from alphagov/stub-staging-sms-providers
Stub out SMS providers on staging for the perf tests
2019-04-24 13:33:49 +01:00
Rebecca Law
bf42fa0c49 Use the csv.reader instead. 2019-04-24 11:56:44 +01:00
Alexey Bezhan
7520cc46de Stub out SMS providers on staging for the perf tests
This points MMG and Firetext on staging to a stub service run on
PaaS to avoid text message costs during the load test.
2019-04-24 11:37:41 +01:00
Alexey Bezhan
dc904f8742 Add a make target to download statsd_exporter binary
Downloads a linux binary from Github and puts it in the scripts folder.
This should add it to the package CI uploads to CF during deploy.

Unfortunately, since out .cfignore is a symlink to .gitignore we
can't add the file to be ignored by git, so it shows up in untracked
if `cf-deploy` is run locally.
2019-04-24 11:25:38 +01:00
Rebecca Law
c989310ac5 New command to populate the go live user and date from the beta partners spreadsheet. 2019-04-24 10:56:50 +01:00
Rebecca Law
d8ea30f414 Merge pull request #2465 from alphagov/go-live-user-id-for-services
Go live data recorded for service
2019-04-18 15:02:21 +01:00
Rebecca Law
cc17ba3ad9 Merge pull request #2455 from alphagov/pyup-scheduled-update-2019-04-10
Scheduled weekly dependency update for week 14
2019-04-17 13:50:42 +01:00
Rebecca Law
55b2fd6d57 Fix the migration file conflicts 2019-04-17 13:39:49 +01:00
Rebecca Law
55a65a22a9 Added test for go_live_user and go_live_at in the response for get_service_by_id 2019-04-17 13:20:44 +01:00
Rebecca Law
60b4970d7f Add new columns to services and services_history.
go_live_user_id: is the user that requested the service to go live
go_live_at: is the DateTime the service went live.

There will be a data migration from the beta partners spreadsheet to back fill the data.
2019-04-17 13:20:44 +01:00
Rebecca Law
0730f9d9e1 Merge branch 'master' into pyup-scheduled-update-2019-04-10 2019-04-17 11:37:47 +01:00
Rebecca Law
b30969dbcf Update requirements.txt 2019-04-17 11:35:42 +01:00
Chris Hill-Scott
96bb9e19f9 Merge pull request #2385 from alphagov/raise-if-fail-to-write-history
Raise exception if history can’t be written
2019-04-17 11:17:49 +01:00
Chris Hill-Scott
2c57e4a4f6 Merge pull request #2445 from alphagov/remove-domains-from-branding-migration
Remove domain columns from branding table
2019-04-17 11:02:27 +01:00
Chris Hill-Scott
916d4e2860 Remove domain columns from branding table
This relationship is via the `Organisation` now; we don’t use this
column to fudge a relationship based on the user’s email address and the
matching something in these columns.
2019-04-15 16:14:41 +01:00
Chris Hill-Scott
8929911ad8 Merge pull request #2444 from alphagov/remove-domains-from-branding
Stop updating or reading domain columns on branding table
2019-04-15 15:35:09 +01:00
Rebecca Law
6ccb242107 Merge pull request #2458 from alphagov/remove-unused-method
Remove unused method.
2019-04-15 12:11:00 +01:00
Rebecca Law
d5ae740f83 Merge pull request #2459 from alphagov/update-response-for-duplicate-domains
The previous PR added a error handler for the wrong db exception.
2019-04-15 12:10:44 +01:00
Leo Hemsted
b38199c666 unpin patch version of python
3.5.5 is removed in the next version of python-buildpack
2019-04-12 16:49:58 +01:00
Chris Hill-Scott
c6447c111d Don’t let admin specify branding for new service
It doesn’t need to because the API code works it all out now by looking
at the organisation.
2019-04-12 16:42:50 +01:00