Commit Graph

1824 Commits

Author SHA1 Message Date
Rebecca Law
eaf5cbb868 Add labels to query so that the named tuples can be referenced later.
Remove unnecessary function
2017-07-25 11:43:41 +01:00
Rebecca Law
91f29517eb Merge branch 'master' into month-billing-table 2017-07-24 15:17:36 +01:00
Rebecca Law
3e2b8190b9 - Added a scheduled task to create or update billing for the month, yesterday is used to calculate the start and end date for the month.
- The new task has not been added to the beat application yet.
- Added an updated_at column to the monthly billing table, we may want to only calculate from the last updated date rather than the entire month.
2017-07-24 15:13:18 +01:00
Leo Hemsted
6059dcfe11 Merge pull request #1093 from alphagov/queue-split-part-2
[2/3] send to new queues
2017-07-24 15:11:37 +01:00
Leo Hemsted
614880f6d9 send to send-sms-tasks and send-email-tasks instead of send-tasks 2017-07-21 13:49:37 +01:00
Rebecca Law
0e3d84c91c Merge pull request #1084 from alphagov/add-logging-to-sms-drs
Add logging to SMS delivery receipts
2017-07-21 12:29:45 +01:00
Leo Hemsted
cfbf7ba2d1 Merge pull request #1092 from alphagov/old-queue-split
[1/3] add separate send-sms and send-email queues
2017-07-21 11:31:48 +01:00
Rebecca Law
d4bbca2592 Fix for simulated notifications.
When a post is made for a simulated number the id is empty in the notificaiton object that we return.
This fixes that.
2017-07-20 17:56:51 +01:00
Leo Hemsted
4d33040653 add separate send-sms and send-email queues
we're reading from those two queues as well as teh existing send queue,
however for now we don't send anything to them
2017-07-20 16:19:38 +01:00
Leo Hemsted
6c61a3fc2a Revert celery4
Revert the following three pull requests:
https://github.com/alphagov/notifications-api/pull/1085
https://github.com/alphagov/notifications-api/pull/1086
https://github.com/alphagov/notifications-api/pull/1088

celery 4.0.2 looked promising, however, on staging under mild load
(5/sec api calls) the performance was actually worse than 3.1.25
2017-07-19 15:17:19 +01:00
Rebecca Law
301339e689 Merge branch 'master' into add-logging-to-sms-drs 2017-07-19 14:30:51 +01:00
Rebecca Law
9400988d72 Monthly billing - part 1
This is still a work in progress but it would be good to get some eyes on it.
This commit includes creating and updating a row in the monthly billing table and a method to fetch the results.
There is a command to populate the monthly billing for a service and month so we can try it out.
The total cost at the moment are wrong, they do not take into account the free allowance - see notes below about adding that to the table.
Left to do:
create a nightly task to run to update the monthly totals.
create an endpoint to return the yearly billing, the current day will need to be calculated on the fly and added to the totals.
Add the free allowance into the total costs.
2017-07-18 18:21:35 +01:00
Rebecca Law
4b05c32b62 Create a new table to warehouse the monthly billing numbers 2017-07-13 17:22:11 +01:00
Leo Hemsted
6d1ec3eb44 Merge pull request #1086 from alphagov/celery4
remove `sqs` from region - this is only needed in boto2
2017-07-13 13:18:58 +01:00
Leo Hemsted
d577855eb3 remove sqs from region - this is only needed in boto2
We don't use boto2 on the api anymore, not since celery 4.0.2

Note - if you run locally with boto2 still installed you'll see errors
that complain about things like:

boto.exception.SQSError: SQSError: 403 Forbidden
<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Credential should be scoped to a valid region, not 'queue'. </Message><Detail/></Error><RequestId>52207ca4-9131-58cb-89ae-2d45f06623a3</RequestId></ErrorResponse>

If so, make sure boto2 is completely uninstalled.
2017-07-13 13:16:28 +01:00
Leo Hemsted
2aae0f9300 Merge pull request #1085 from alphagov/celery4
Celery 4 (attempt 2)
2017-07-13 11:54:32 +01:00
Leo Hemsted
3331491ef1 add pycurl to requirements
it's needed by kombu 4.0.2.

Also moved import about
2017-07-12 17:17:02 +01:00
Rebecca Law
49d1f52aef Fix code style 2017-07-12 15:49:43 +01:00
Rebecca Law
d18ce47114 Make sure we don't log the phone number 2017-07-12 15:32:59 +01:00
Rebecca Law
6fb4e16067 Added logging to show the entire form posted to us by the SMS client providers.
This can be useful information when debugging what happened to a notificaiton.
Recently there was a discrepancy between the failure type used by each provider for a particular number, this logging would have helped.
2017-07-12 14:19:39 +01:00
Leo Hemsted
1a03248317 temp fix to sort out circular imports 2017-07-12 13:02:19 +01:00
Martyn Inglis
0e9e8955f7 Finished celery refactor - set up config for queue prefix
LEO notes: Also made sure the Test BROKER_URL is preserved so that
tests warn you when celery isn't mocked out
2017-07-12 12:37:18 +01:00
Martyn Inglis
786adb5d71 Move Queuenames in with the celery code, revamp config to allow move to celery 4.x 2017-07-12 12:01:52 +01:00
Ken Tsang
0c77f2d010 Refactored dao_update_organisation 2017-07-12 12:01:50 +01:00
Ken Tsang
106d59006b Removed dao_update_organisation 2017-07-12 12:01:50 +01:00
Ken Tsang
af9091f207 Refactor tests to use admin_request 2017-07-12 12:01:50 +01:00
Ken Tsang
3b1f229384 Renamed migration script and refactor code 2017-07-12 12:01:50 +01:00
Ken Tsang
b814c5ff26 Refactor organisation rest to remove marshmallow 2017-07-12 12:01:50 +01:00
Ken Tsang
76493a209c Added POST for organisation rest 2017-07-12 12:01:50 +01:00
Ken Tsang
2f8bc0feae Add create and update fns for organisation dao 2017-07-12 12:01:50 +01:00
Ken Tsang
a6df96213b Update model and migration script 2017-07-12 12:01:50 +01:00
Ken Tsang
f76962ad4d Add versioning to Organisations 2017-07-12 12:01:50 +01:00
Ken Tsang
37df051c32 Updated Organisations DAO 2017-07-12 12:01:50 +01:00
Ken Tsang
c743e52fe8 Add organisations model and create dao 2017-07-12 12:01:50 +01:00
Martyn Inglis
e0106eb1be hacked celery4.0.2 in. Runs and works
- note though this version of master I branched had split head on sqlalchemey. This needs a new master merge to fix
2017-07-12 11:58:04 +01:00
Rebecca Law
a01163d719 Merge pull request #1077 from alphagov/drop-template_stats-table
Remove the archived table template_statistics.
2017-07-12 10:41:32 +01:00
Leo Hemsted
7f883f1355 don't store non-strings to os.environ
in tests, we were replacing os.environ with a basic dict so that
we didn't overwrite the contents of the real environment during tests.
However, os.environ doesn't accept non-str values, so this commit
changes the fixture so that it asserts all values set are strings.
We needed to change how we store ip whitelist stuff in the env because
of this.
2017-07-11 15:41:44 +01:00
venusbb
d3db4a6a01 Added a unit test to validate traffic from more than 3 hops 2017-07-11 09:59:41 +01:00
venusbb
5d57189187 changed name of test fixture 2017-07-11 09:50:09 +01:00
venusbb
226ae5784b reading ip address 3rd from the back rather than the first one 2017-07-10 17:33:13 +01:00
venusbb
50d01d18c5 reading ip address 3rd from the back rather than the first one 2017-07-10 17:25:38 +01:00
venusbb
5089a4d53b retrieve sms ip whitelist from credentials on paas 2017-07-10 17:03:43 +01:00
Imdad Ahad
44c5831f3d Remove old status schema references 2017-07-10 15:50:57 +01:00
Rebecca Law
53507314ea Merge branch 'master' into drop-template_stats-table 2017-07-10 15:35:49 +01:00
Rebecca Law
8a01a76e33 Remove the archived table template_statistics. The last time the table we updated was August 30 2016, it's safe to say we are done with it.
I updated the InboundSms and TemplateRedacted model to include an index in the db.
Dropped service_permissions.updated_at column since we are not auditting the table
2017-07-10 14:43:46 +01:00
Imdad Ahad
a9c1338873 Remove Notification, NotificationHistory status labels:
Replace labels by adding a key kwarg in the model for status.

We still need this as sqlalchemy attmempts to look for `notification_status`
on the model (Notification/NotificationHistory). To achieve true ORM mapping
(map status -> notification_status) we need the key kwarg.

More here:
http://docs.sqlalchemy.org/en/latest/core/metadata.html#sqlalchemy.schema.Column#key
2017-07-10 14:09:30 +01:00
Imdad Ahad
c8c47f44a9 Don't update older status in timeout notifications task 2017-07-10 14:09:30 +01:00
Imdad Ahad
ac9eb29707 Revert "Revert "1/4 Stop updating old Notification status column"" 2017-07-10 14:09:30 +01:00
Imdad Ahad
ae388a3cfc Make Noti and NotiHis status column nullable:
This is in preparation for deprecating this old column
2017-07-10 12:09:02 +01:00
Rebecca Law
50dbfec383 Merge pull request #1059 from alphagov/update-purge_function
Refactor order of delete statements
2017-07-07 15:32:29 +01:00