Commit Graph

3081 Commits

Author SHA1 Message Date
Leo Hemsted
49bbd1a29b Merge pull request #1090 from alphagov/revert-celery-4
Revert celery4
2017-07-19 16:57:31 +01:00
Leo Hemsted
6c61a3fc2a Revert celery4
Revert the following three pull requests:
https://github.com/alphagov/notifications-api/pull/1085
https://github.com/alphagov/notifications-api/pull/1086
https://github.com/alphagov/notifications-api/pull/1088

celery 4.0.2 looked promising, however, on staging under mild load
(5/sec api calls) the performance was actually worse than 3.1.25
2017-07-19 15:17:19 +01:00
Imdad Ahad
92755c3417 Merge pull request #1078 from alphagov/imdad-migration-drop-old-noti-status
3/3 Drop old notification status column
2017-07-18 11:52:09 +01:00
Leo Hemsted
965755bf13 Merge pull request #1088 from alphagov/kombu-bump
update kombu version to prevent 100% CPU usage
2017-07-17 14:42:45 +01:00
Leo Hemsted
bfdf9bc9a0 update kombu version to prevent 100% CPU usage
This pulls in the most recent version of celery/kombu/master - it
prevents situations where the celery MainProcess would consume 100%
CPU while not distributing any tasks to the workers.

Note: If, locally, you already have kombu==4.0.2, you'll have to run
pip install --upgrade https://github.com/celery/kombu/zipball/b2f21289
to get the new version.
2017-07-17 14:21:01 +01:00
Leo Hemsted
6d1ec3eb44 Merge pull request #1086 from alphagov/celery4
remove `sqs` from region - this is only needed in boto2
2017-07-13 13:18:58 +01:00
Leo Hemsted
d577855eb3 remove sqs from region - this is only needed in boto2
We don't use boto2 on the api anymore, not since celery 4.0.2

Note - if you run locally with boto2 still installed you'll see errors
that complain about things like:

boto.exception.SQSError: SQSError: 403 Forbidden
<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Credential should be scoped to a valid region, not 'queue'. </Message><Detail/></Error><RequestId>52207ca4-9131-58cb-89ae-2d45f06623a3</RequestId></ErrorResponse>

If so, make sure boto2 is completely uninstalled.
2017-07-13 13:16:28 +01:00
Leo Hemsted
2aae0f9300 Merge pull request #1085 from alphagov/celery4
Celery 4 (attempt 2)
2017-07-13 11:54:32 +01:00
Leo Hemsted
8679fc35d1 pin kombu to fix SQS issues
Kombu is a library that celery uses under the hood. Kombu is pinned to
a specific commit that brings in fixes for SQS - see
https://github.com/celery/kombu/pull/693. Kombu v4.0.2 (which ships
with celery v4.0.2) doesn't work with SQS due to problems with their
use of boto2, so Kombu migrated to boto3 - We're waiting for that to
get a release version, and then to get a new version of celery that
pulls that in. Until that point, we should override the kombu version
to get these SQS fixes.

Additionally, we don't need boto2 any more :)
2017-07-13 11:31:45 +01:00
Leo Hemsted
3331491ef1 add pycurl to requirements
it's needed by kombu 4.0.2.

Also moved import about
2017-07-12 17:17:02 +01:00
Imdad Ahad
555dac76a8 Remove old notification status column 2017-07-12 16:24:18 +01:00
Leo Hemsted
4ced59f7d3 remove pycurl from reqs 2017-07-12 16:01:22 +01:00
Leo Hemsted
1a03248317 temp fix to sort out circular imports 2017-07-12 13:02:19 +01:00
kentsanggds
eb8cf26c80 Merge pull request #1073 from alphagov/ken-update-organisation-dao-model-rest
Update organisation dao model rest
2017-07-12 12:57:06 +01:00
Martyn Inglis
0e9e8955f7 Finished celery refactor - set up config for queue prefix
LEO notes: Also made sure the Test BROKER_URL is preserved so that
tests warn you when celery isn't mocked out
2017-07-12 12:37:18 +01:00
Ken Tsang
f53c0cfb8a Rename migration script 2017-07-12 12:05:03 +01:00
Martyn Inglis
786adb5d71 Move Queuenames in with the celery code, revamp config to allow move to celery 4.x 2017-07-12 12:01:52 +01:00
Ken Tsang
0c77f2d010 Refactored dao_update_organisation 2017-07-12 12:01:50 +01:00
Ken Tsang
106d59006b Removed dao_update_organisation 2017-07-12 12:01:50 +01:00
Ken Tsang
1ca59c4b44 Renamed migration script 2017-07-12 12:01:50 +01:00
Ken Tsang
af9091f207 Refactor tests to use admin_request 2017-07-12 12:01:50 +01:00
Ken Tsang
3b1f229384 Renamed migration script and refactor code 2017-07-12 12:01:50 +01:00
Ken Tsang
b814c5ff26 Refactor organisation rest to remove marshmallow 2017-07-12 12:01:50 +01:00
Ken Tsang
12e6cd0a8f Add test for missing data 2017-07-12 12:01:50 +01:00
Ken Tsang
76493a209c Added POST for organisation rest 2017-07-12 12:01:50 +01:00
Ken Tsang
2f8bc0feae Add create and update fns for organisation dao 2017-07-12 12:01:50 +01:00
Ken Tsang
a6df96213b Update model and migration script 2017-07-12 12:01:50 +01:00
Ken Tsang
f76962ad4d Add versioning to Organisations 2017-07-12 12:01:50 +01:00
Ken Tsang
37df051c32 Updated Organisations DAO 2017-07-12 12:01:50 +01:00
Ken Tsang
c743e52fe8 Add organisations model and create dao 2017-07-12 12:01:50 +01:00
Martyn Inglis
e0106eb1be hacked celery4.0.2 in. Runs and works
- note though this version of master I branched had split head on sqlalchemey. This needs a new master merge to fix
2017-07-12 11:58:04 +01:00
Rebecca Law
a01163d719 Merge pull request #1077 from alphagov/drop-template_stats-table
Remove the archived table template_statistics.
2017-07-12 10:41:32 +01:00
Leo Hemsted
19481cda02 Merge pull request #1081 from alphagov/ip-fix
don't store non-strings to os.environ
2017-07-11 16:10:17 +01:00
Leo Hemsted
7f883f1355 don't store non-strings to os.environ
in tests, we were replacing os.environ with a basic dict so that
we didn't overwrite the contents of the real environment during tests.
However, os.environ doesn't accept non-str values, so this commit
changes the fixture so that it asserts all values set are strings.
We needed to change how we store ip whitelist stuff in the env because
of this.
2017-07-11 15:41:44 +01:00
Venus Bailey
2fde4b2c80 Merge pull request #1079 from alphagov/sms_whitelist
retrieve sms ip whitelist from credentials on paas
2017-07-11 14:05:52 +01:00
venusbb
d3db4a6a01 Added a unit test to validate traffic from more than 3 hops 2017-07-11 09:59:41 +01:00
venusbb
5d57189187 changed name of test fixture 2017-07-11 09:50:09 +01:00
venusbb
6b650c6d96 changed name of test fixture 2017-07-11 09:33:56 +01:00
venusbb
226ae5784b reading ip address 3rd from the back rather than the first one 2017-07-10 17:33:13 +01:00
venusbb
50d01d18c5 reading ip address 3rd from the back rather than the first one 2017-07-10 17:25:38 +01:00
venusbb
5089a4d53b retrieve sms ip whitelist from credentials on paas 2017-07-10 17:03:43 +01:00
Imdad Ahad
c0af82ebcb Merge pull request #1065 from alphagov/revert-1064-revert-1057-imdad-feat-stop-updating-old-noti-status
2/3 Stop updating old Notification status column
2017-07-10 16:39:48 +01:00
Imdad Ahad
44c5831f3d Remove old status schema references 2017-07-10 15:50:57 +01:00
Rebecca Law
9bf1efd16a Fix merge conflict with db migration script 2017-07-10 15:44:01 +01:00
Rebecca Law
53507314ea Merge branch 'master' into drop-template_stats-table 2017-07-10 15:35:49 +01:00
Rebecca Law
8a01a76e33 Remove the archived table template_statistics. The last time the table we updated was August 30 2016, it's safe to say we are done with it.
I updated the InboundSms and TemplateRedacted model to include an index in the db.
Dropped service_permissions.updated_at column since we are not auditting the table
2017-07-10 14:43:46 +01:00
Imdad Ahad
a9c1338873 Remove Notification, NotificationHistory status labels:
Replace labels by adding a key kwarg in the model for status.

We still need this as sqlalchemy attmempts to look for `notification_status`
on the model (Notification/NotificationHistory). To achieve true ORM mapping
(map status -> notification_status) we need the key kwarg.

More here:
http://docs.sqlalchemy.org/en/latest/core/metadata.html#sqlalchemy.schema.Column#key
2017-07-10 14:09:30 +01:00
Imdad Ahad
c8c47f44a9 Don't update older status in timeout notifications task 2017-07-10 14:09:30 +01:00
Imdad Ahad
ac9eb29707 Revert "Revert "1/4 Stop updating old Notification status column"" 2017-07-10 14:09:30 +01:00
Imdad Ahad
4b2388acaa Merge pull request #1076 from alphagov/imdad-migration-make-old-notification-status-nullable
1/3 Make Notification._status_enum and NotificationHistory._status_enum nullable
2017-07-10 13:47:31 +01:00