Commit Graph

2360 Commits

Author SHA1 Message Date
Chris Hill-Scott
c2999ed5ad Use pycodestyle instead of PEP8
PEP8 was renamed to pycodestyle; this issue explains why:
https://github.com/PyCQA/pycodestyle/issues/466

This commit changes our tests to use pycodestyle instead of pep8.

It also means:
- making a couple of whitespace changes to appease the linter
- disabling warnings for bare `Except`s (ie `Except` instead of `Except
  ValueError`) – this seems like a sensible thing to catch but I’m not
  going to make meaningful code changes in this commit
2017-02-01 09:19:32 +00:00
Leo Hemsted
855a412c90 Merge pull request #812 from alphagov/plat-adm-dates
make sure we convert strings to dates before we use them
2017-01-31 16:24:30 +00:00
Leo Hemsted
cd47dbd36b make sure we convert strings to dates before we use them
fixes 500 on platform admin page
2017-01-31 16:12:46 +00:00
Chris Hill-Scott
d0b467b2fb Merge pull request #807 from alphagov/monthly-service-usage
Add endpoint for breakdown of activity by month
2017-01-31 16:04:32 +00:00
Rebecca Law
dab89969cc If the service is inactive at the time of sending the notification to the provider, the notification is marked as technical-failure.
Removed the /service/<id>/deactivate endpoint, now using /service/<id>/archive
2017-01-31 14:28:25 +00:00
Chris Hill-Scott
45b505f379 Split out handling of BST queries for reuse 2017-01-31 14:16:35 +00:00
Chris Hill-Scott
8381a5e11c Ensure that monthly service usage groups by BST 2017-01-31 14:16:35 +00:00
Chris Hill-Scott
56ba653f48 Add endpoint for breakdown of activity by month
This endpoint will eventualy replace the weekly breakdown one. By month
for a given financial year is better, because it gives us consistency
with the breakdown of financial usage (and eventually consistency with
the template usage).

The code to do this is a bit convoluted, in order to fill out the counts
for months and statuses where we don’t have notifications.

This will make the admin side of this easier, because we can rely on
there always being numbers available. The admin side will deal with
summing the statuses (eg `temporary-failure` > `failed`) because this
is presentational.

This commit also modifies the usage count to use `.between()` for
consistency.
2017-01-31 14:16:34 +00:00
Rebecca Law
fde3216f9e Check the service is active before sending the notification to the provider. 2017-01-31 13:53:13 +00:00
Leo Hemsted
3113f49271 Merge pull request #810 from alphagov/letter-fix
fix api statistics to account for letters
2017-01-31 12:29:35 +00:00
bandesz
e2a466932a Put Jenkinsfile milestones before the prompts 2017-01-31 12:18:10 +00:00
Leo Hemsted
0d089a383e fix tests in service/test_rest.py 2017-01-31 12:06:25 +00:00
Rebecca Law
ef0f708da0 Merge branch 'master' into check-service-is-active 2017-01-31 12:00:30 +00:00
Rebecca Law
e2e5cb6866 Merge pull request #808 from alphagov/disable-service
Suspend/resume service feature
2017-01-31 11:56:22 +00:00
Rebecca Law
539ebdce70 This is what I meant to test 2017-01-31 11:45:35 +00:00
Rebecca Law
907a1e91ac Removed the need for 2 return statements.
Update tests to fix copy/paste errors, use freezetime to ensure there is never a random failing test.
2017-01-31 11:33:36 +00:00
Leo Hemsted
3cd8605e11 fix api statistics to account for letters 2017-01-31 11:32:53 +00:00
imdadahad
0957520735 Merge pull request #809 from alphagov/fix-perf-platform-stats-job
Fix perf platform stats job
2017-01-31 11:23:03 +00:00
Rebecca Law
fe64b565a0 Create job now checks that the service is active before creating job.
Need to check what message and status code we should use for this case. Probably will create a InactiveServiceRequest exception.
2017-01-31 11:11:33 +00:00
Imdad Ahad
53b6cdcfab Only send stats from celery task if active + refactor tests 2017-01-30 18:24:18 +00:00
Imdad Ahad
c811f1b6c6 Init the perf platform client, add logs and refactor payload methods 2017-01-30 18:24:06 +00:00
Rebecca Law
87556687ab This is the first PR in the story to create a suspend service feature.
- Renaming /service/<id>/deactivate to /service/<id>/archive to match language on the UI.
  - Will need to update admin before deleting the deactive service method
- Created dao and endpoint methods to suspend and resume a service.
- I confirm the use of suspend and resume with a couple people on the team, seems to be the right choice.

The idea is that if you archive a service there is no coming back from that.
To suspend a service is marking it as inactive and revoking the api keys. To resume a service is to mark the service as active, the service will need to create new API keys.
It makes sense that if a service is under threat that the API keys should be renewed.

The next PR will update the code to check that the service is active before sending the request or allowing any actions by the service.
2017-01-30 16:32:44 +00:00
Leo Hemsted
1650fb0807 Merge pull request #797 from alphagov/persist-letters
Persist letters
2017-01-30 14:24:38 +00:00
Leo Hemsted
0f7d4548b0 Merge pull request #806 from alphagov/add-logging-for-ses-callback
Small change to include the reference that SES send us on the callback.
2017-01-30 14:24:07 +00:00
Rebecca Law
c6eb284218 Small change to include the reference that ses send us on the callback. 2017-01-30 14:09:30 +00:00
Rebecca Law
1d734d3b28 Merge pull request #805 from alphagov/fix-duplicate-test
Fix duplicate test name
2017-01-30 09:40:41 +00:00
imdadahad
eec7564475 Merge pull request #804 from alphagov/feat-add-perf-platform-client-and-job
Add client and job to update the performance platform daily
2017-01-27 17:03:38 +00:00
Imdad Ahad
cc7bf45766 Move notification retrieval logic into perf platform client and dont filter by status 2017-01-27 16:39:01 +00:00
Rebecca Law
090ecb0de1 Revert local change to config 2017-01-27 16:09:11 +00:00
Rebecca Law
c87d0a37f3 Refactor the get_midnight functions to return the date in UTC but for the localised time.
So in June when we are in BST June 16, 00:00 BST => June 15 23:00 UTC
2017-01-27 15:57:25 +00:00
Rebecca Law
4e4afc5d0b Rename test with duplicate name 2017-01-27 15:21:59 +00:00
Rebecca Law
fba3332488 Merge pull request #802 from alphagov/return-200-for-duplicate-callback
Return success of the callback is a duplicate or for an id that does not exist
2017-01-27 14:02:56 +00:00
bandesz
568d93e67f Update branch to master in Jenkinsfile 2017-01-27 13:57:15 +00:00
minglis
2057fcdbdd Merge pull request #801 from alphagov/remove-task-from-research-mode-process
Change to research mode.
2017-01-27 13:30:41 +00:00
Imdad Ahad
ac1ca608dd Add performance platform url directly in config (not a secret) 2017-01-27 13:03:04 +00:00
Imdad Ahad
b04c524bc3 Add celery task to run daily at 00:30 that sends notification counts to performance platform 2017-01-27 12:30:56 +00:00
Imdad Ahad
b6cc5b00ba Add dao method to get sms and email notification counts for a date range and previous day 2017-01-27 12:22:36 +00:00
Imdad Ahad
781bef557a Add date helper methods to be used as date range filters for notification counts 2017-01-27 12:21:28 +00:00
Imdad Ahad
ed52f41039 Add, init and config performance platform client to prepare/upload daily notification stats 2017-01-27 12:21:08 +00:00
Rebecca Law
27b6aa736c Update info log to waring.
Removed duplicate test
2017-01-27 12:20:54 +00:00
Chris Hill-Scott
355f650f86 Merge pull request #803 from alphagov/filter-global-usage-financial-year
Allow filtering of billing info by financial year
2017-01-27 11:58:28 +00:00
Chris Hill-Scott
4e60265c58 Add tests for billable unit rest endpoint
Doesn’t do extensive tests through the DAO layer, we have separate tests
elsewhere for that. This just tests the query parameter validation.
2017-01-27 11:47:40 +00:00
Chris Hill-Scott
8ad0236f28 Allow filtering of billing info by financial year
We already filter the usage-by-month query by financial year. When we
show the total usage for a service, we should be able to filter this
by financial year.

Then, when the two lots of data are put side by side, it all adds up.
2017-01-27 11:13:22 +00:00
Rebecca Law
67657ced26 Return success of the callback is a duplicate or for an id that does not exist
This PR changes the response to POST /notifications/sms/<mmg | firetext> from a 400 response to a 200 response.
If we get a callback for a notification more than once or for a notification we log that but we return a 200 success response to the provider.
We have found that there is a situation where the send to provider throws a timeout exception but the provider did get the message, but we still send it to them again.
In which case they send the message twice, and callback for the message twice.
Another case where we may get duplicate callbacks is that the network gave the provider two callbacks meaning they pass those two callbacks onto us.

So it is really difficult to know if we sent to the provider twice or just got two callbacks.

The test_callback has many changes because I took the opportunity to use the client conftest fixture rather than the notify_api fixture.
The only 2 tests really changed are test_mmg_callback_returns_200_when_notification_id_not_found_or_already_updated and test_firetext_callback_returns_200_when_notification_id_not_found_or_already_updated
2017-01-25 14:37:11 +00:00
Martyn Inglis
71289d36db Change to research mode.
Previously research mode created a task to fake the callback from the providers. This meant there is an extra task for each notification than would be generated in a live situation.

This PR changes that, so that a research mode/test key notification calls the callback API rather than make a task to do that .

This ensures that the flow for research mode more closely mimics that of live, and removes a task from the process so we can more accurately test throughput,
2017-01-24 14:24:58 +00:00
Leo Hemsted
70cd3fb335 ensure that the celery workers know about the new db-letter queue
this fixes running locally and on paas, a separate PR is in
notifications-aws to fix work on aws
2017-01-24 12:32:20 +00:00
Leo Hemsted
4f238d241a persist_letter saves address correctly to database
the `to` field stores either the phone number or the email address
of the recipient - it's a bit more complicated for letters, since
there are address lines 1 through 6, and a postcode. In utils, they're
stored alongside the personalisation, and we have to ensure that when
we persist to the database we keep as much parity with utils to make
our work easier. Aside from sending, the `to` field is also used to
show recipients on the front end report pages - we've decided that the
best thing to store here is address_line_1 - which is probably going to
be either a person's name, company name, or PO box number

Also, a lot of tests and test cleanup - I added create_template and
create_notification functions in db.py, so if you're creating new
fixtures you can use these functions, and you won't need to pass
notify_db and notify_db_session around, huzzah!

also removed create param from sample_notification since it's not used
anywhere
2017-01-24 12:32:20 +00:00
Leo Hemsted
c904025ee9 add comprehensive tests for process_row
in the future we can probably remove some of the slower databasey
process_job tests, but that's out of scope for this
2017-01-24 12:22:31 +00:00
Leo Hemsted
542f08d0b5 refactor exception handling code in job tasks code
it's almost entirely duplicated so share it across.

also clean up retrying - `task.retry(...)` raises a
celery.exceptions.Retry object, so you do not need to `raise` its
response. additionally, cleaned up tests around that since raising
Exception and asserting Exception is raised is dangerous as it could
mask actual programming errors
2017-01-24 12:22:31 +00:00
Leo Hemsted
0324ba02ff split process_job up into process_job and process_row
makes tests a bit cleaner, and makes it much easier to test letter
functionality. haven't moved many tests around, just changed a couple of mock calls
2017-01-24 12:22:31 +00:00