darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2026-01-19 17:11:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
4,992 Commits 134 Branches 60 Tags
52aeec22476b1978d06d6fc0e44c790d48923d63
Commit Graph

2464 Commits

Author SHA1 Message Date
venusbb
52aeec2247 bug fix: use date only when comparing unique records rather than datetime 2018-03-29 11:42:31 +01:00
venusbb
cf766313e6 Merge branch 'master' of https://github.com/alphagov/notifications-api into vb-report-tasks 2018-03-28 10:53:45 +01:00
Rebecca Law
63e85e3511 Merge pull request #1806 from alphagov/update-logging 
Update logging for provider tasks.
 2018-03-27 14:04:12 +01:00
Richard Chapman
20179af6e6 Merge pull request #1802 from alphagov/rc_add_virus_scan_error_task 
Added a new task to handle any error cases with the anti-virus
 2018-03-27 14:03:24 +01:00
venusbb
2262db4f62 Database tweak and BST bug fix 2018-03-27 10:37:56 +01:00
Rebecca Law
598539dcb3 Update logging for provider tasks. 
Move the info message before the fetch.
Include the exception in the log message.
 2018-03-26 15:24:21 +01:00
Richard Chapman
8b6d28d3b0 Added a new task to handle any error cases with the anti-virus 
application. If the Anti-virus app fails due to s3 errors or ClamAV
so does not scan (even after retries) the file at all an error needs
to be raised and the notification set to technical-failure.

Files should be moved to a 'folder' a separate one for ERROR and FAILURE.

* Added new letter task to process the error
* Added a new method to letter utils.py to move a file into an error or
failure folder based on the input
* Added tests to test the task and the utils.py method
 2018-03-26 14:18:44 +01:00
Rebecca Law
bdae669b05 Merge pull request #1800 from alphagov/replay-created-emails 
Replay emails and sms that are still in created
 2018-03-26 10:56:55 +01:00
Rebecca Law
acfb1305f4 Fix test 2018-03-26 10:51:38 +01:00
Rebecca Law
f596d17bf2 If a sms or email has not been sent after 4 hours and 15 minutes then put it on the delivery queue. 2018-03-23 15:38:35 +00:00
kentsanggds
5d38989cd9 Merge pull request #1797 from alphagov/ken-test-key-antivirus 
Add antivirus check on precompiled letters sent with test key
 2018-03-23 15:04:08 +00:00
Ken Tsang
1a9bc2a5cf Move test letters to test letters bucket without date folder name 2018-03-23 14:59:48 +00:00
Ken Tsang
0ee5c33084 Add antivirus check on precompiled letters sent with test key 
- precompiled PDFs sent by test key uploaded to scan bucket
- set status to VIRUS-SCAN-FAILED for pdfs failing virus scan rather than PERMANENT-FAILURE
- Make call to AV app for precompiled letters sent via a test key, and set notification status to PENDING-VIRUS-SCAN
 2018-03-23 12:04:37 +00:00
Chris Waszczuk
9bf739007a Merge pull request #1790 from alphagov/update-paginated-inbound-sms-method 
Update inbound sms method
 2018-03-23 11:25:47 +00:00
chrisw
ac50250ad7 update paginated inbound sms method 2018-03-23 11:09:51 +00:00
kentsanggds
832c350b69 Merge pull request #1789 from alphagov/ken-fix-no-sent_by 
Set sent_by if not set during sms provider callback
 2018-03-23 09:52:26 +00:00
Ken Tsang
c272d023c4 Lowercase the client name when sent_by is empty 2018-03-22 16:12:33 +00:00
Ken Tsang
ffa093d8c7 Set sent_by if not set during sms provider callback 
- refactored argument reference to provider_reference to make it clearer
 2018-03-21 18:11:10 +00:00
kentsanggds
5dc0248043 Merge pull request #1783 from alphagov/ken-process-antivirus 
Send task to antivirus app and process antivirus callbacks
 2018-03-21 16:39:55 +00:00
venusbb
cf019864e2 Preserve 'unknown' in ft_billing for sms if the provider is not known. 2018-03-21 09:50:34 +00:00
Rebecca Law
81f9c06e06 Merge pull request #1781 from alphagov/only-use-data-on-queue-not-from-db 
Remove unused code in send_delivery_status_to_service task
 2018-03-20 16:13:42 +00:00
Rebecca Law
0798154fa2 Optimize the query used to return the services and todays notification totals. 
By changing the created_at filter to use a specific date range I found a significant improvement to the queries performance.
The unit test needed to change because now were are returning todays date as BST the local timezone.

Query plan before

Merge Left Join  (cost=1226133.76..1226143.77 rows=1753 width=70) (actual time=5800.160..5801.657 rows=1849 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.205..2.631 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.011..1.156 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=1225980.77..1225980.99 rows=86 width=40) (actual time=5797.949..5797.984 rows=198 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=1225976.29..1225978.01 rows=86 width=40) (actual time=5797.682..5797.823 rows=198 loops=1)
->  HashAggregate  (cost=1225976.29..1225977.15 rows=86 width=48) (actual time=5797.681..5797.747 rows=198 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Seq Scan on notifications  (cost=0.00..1220610.86 rows=536543 width=48) (actual time=0.064..5482.975 rows=643799 loops=1)
Filter: (((key_type)::text <> 'TEST'::text) AND (date(created_at) = '2018-03-20'::date))
Rows Removed by Filter: 6804774
Planning time: 1.106 ms
Execution time: 5802.130 ms

Query plan after
Merge Left Join  (cost=953378.30..953388.30 rows=1753 width=70) (actual time=2380.144..2382.499 rows=1852 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.944..3.570 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.006..1.294 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=953225.31..953225.53 rows=86 width=40) (actual time=2377.194..2377.262 rows=201 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=953220.83..953222.55 rows=86 width=40) (actual time=2376.797..2377.034 rows=201 loops=1)
->  HashAggregate  (cost=953220.83..953221.69 rows=86 width=48) (actual time=2376.795..2376.905 rows=201 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Bitmap Heap Scan on notifications  (cost=29883.14..947856.24 rows=536459 width=48) (actual time=270.061..1887.754 rows=644735 loops=1)
Recheck Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Rows Removed by Index Recheck: 947427
Filter: ((key_type)::text <> 'TEST'::text)
Heap Blocks: exact=40882 lossy=186483
->  Bitmap Index Scan on ix_notifications_created_at  (cost=0.00..29749.02 rows=536459 width=0) (actual time=258.631..258.631 rows=644849 loops=1)
Index Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Planning time: 0.548 ms
Execution time: 2383.485 ms
 2018-03-20 15:48:32 +00:00
Ken Tsang
8391353113 Update tasks in line with AV app 2018-03-20 14:56:42 +00:00
Ken Tsang
d9494dbc97 Refactor tests after rebase 2018-03-20 14:56:26 +00:00
Ken Tsang
65733a30a1 Add send_task to antivirus app for scanning precompiled letters 2018-03-20 10:12:59 +00:00
Ken Tsang
b1ac580e0a Add celery tasks to handle virus scan passing and failing 2018-03-20 10:12:59 +00:00
Ken Tsang
0c102f0727 Update letter utils for methods to handle virus process 
- add function to get reference from filename
- add function to move pdf from scan folder to process folder
- add function to delete pdfs from scan bucket for failed virus scans
 2018-03-20 10:12:59 +00:00
Ken Tsang
8e78c5f286 Refactor letter test 
- move letter tests from test_post_notifications.py to test_post_letter_notifications.py
 2018-03-20 10:12:59 +00:00
Ken Tsang
8733d84e75 Upload precompiled letter pdfs to letters-scan bucket 2018-03-20 10:11:36 +00:00
Ken Tsang
30e371fa4c Set precompiled letters to pending virus check initially 2018-03-20 10:11:36 +00:00
Rebecca Law
ee46803a12 The send_delivery_status_to_service task was refactor to take the details of the notification and service api callback such that the task no longer needed to go to the database to provide the status update. 
This PR removes the code that is no longer used. This extra step was necessary to keep the tasks backward compatible.
 2018-03-19 17:38:20 +00:00
Rebecca Law
22c296b0ef Merge pull request #1780 from alphagov/send-service-callback-if-sent_at_is-None 
Send service callback if sent at is none
 2018-03-19 16:32:15 +00:00
Rebecca Law
cd2d85f2a3 Updates after code review. 
- Remove print
- Update exception message.
 2018-03-19 14:08:38 +00:00
Venus Bailey
a810bb4e40 Merge pull request #1771 from alphagov/vb-billing-periodic 
Fine tune reporting model and create task for migrating data [#155511547]
 2018-03-19 09:17:39 +00:00
Rebecca Law
0dc50190b2 Throw an exception whenever we updated a notification to technical failure. 
If this is happening we want to know about it.
 2018-03-16 17:18:44 +00:00
Rebecca Law
c9477a7400 When a notification is timed out in the scheduled task that may happen because the notification has not been sent. 
Which means the sent_at date for the notification could be empty causing the service callback to fail.

- Allow code to work if notification.sent_at or updated_at is None
- Update calls to send_delivery_status_to_service to send the data encrypted so that the task does not need to use the db.
 2018-03-16 14:47:56 +00:00
Rebecca Law
e78549ac9a Added test for all notification types 2018-03-16 14:32:27 +00:00
Rebecca Law
a308815793 We had an exception today caused by a slow running query. 
This PR will optimize this query to use a more efficient index.

- Add notification_type to the dao_get_last_template_usage to optimize the query.
- Tested and analyzed query on production database with very significant results.
 Before:
                                                                                 QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..1711.35 rows=1 width=935) (actual time=21186.053..21186.053 rows=0 loops=1)
   ->  Index Scan Backward using ix_notifications_created_at on notifications  (cost=0.43..4607493.80 rows=2693 width=935) (actual time=21186.052..21186.052 rows=0 loops=1)
         Filter: (((key_type)::text <> 'test'::text) AND (template_id = 'xxxxxx'::uuid))
         Rows Removed by Filter: 8244071
 Planning time: 0.112 ms
 Execution time: 21186.082 ms
 After:
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Limit  (cost=5323.10..5323.10 rows=1 width=935)
   ->  Sort  (cost=5323.10..5323.74 rows=258 width=935)
         Sort Key: created_at DESC
         ->  Index Scan using ix_notifications_template_id on notifications  (cost=0.56..5321.81 rows=258 width=935)
               Index Cond: (template_id = 'xxxxx'::uuid)
               Filter: (((key_type)::text <> 'test'::text) AND (notification_type = 'sms'::notification_type))
 Planning time: 1.102 ms
 Execution time: 0.584 ms
 2018-03-16 14:12:59 +00:00
Richard Chapman
2a3095d5c5 Merge pull request #1770 from alphagov/Update_notifification_response_for_precompiled 
Update notification response for precompiled
 2018-03-16 14:00:39 +00:00
venusbb
7e2947790f merged master and up migration version 2018-03-16 10:57:23 +00:00
kentsanggds
62d2b04e0a Merge pull request #1773 from alphagov/ken-preview-precompiled-letters-test-key 
Preview precompiled letters sent using an API test key
 2018-03-16 10:24:58 +00:00
venusbb
bb95a2784f Create schedueled job, fixed tests 2018-03-16 09:22:34 +00:00
Rebecca Law
8251766cc0 Merge pull request #1766 from alphagov/user-orgs-and-services 
User orgs and services
 2018-03-15 16:01:36 +00:00
Ken Tsang
d40bc28f2a Add test-letters bucket for precompiled test keys 
- uses moto to mock out boto3 for testing
 2018-03-14 21:27:07 +00:00
Ken Tsang
d3537cca74 Add upload pdf to test-letters bucket 
- triggered when sending a precompiled letter using a test key
 2018-03-14 18:16:43 +00:00
Ken Tsang
c4e1d56492 Update letter utils to handle precompiled letters sent by test API keys 
- precompiled letters sent using a test key should be put into a test-letters bucket
- also refactored code from test_letter_pdf_tasks
 2018-03-14 18:15:00 +00:00
venusbb
5d7d4a0dd6 Added test for reporting tasks 2018-03-14 17:07:52 +00:00
Rebecca Law
82cc6d6bef As it turns out the s3ftp used to mount the s3 bucket to our ftp server puts the file on s3 more than once. So the SNS topic is triggered more than once. 
We need to deal with this, it's ok when updating a notification status from delivered to delivered. But the DailySortedLetter counts are being doubled.
Adding the file_name to the table as a unique key to get around this issue. It will mean we have multiple rows for each billing_day, but that's ok we can aggregate that.
This will also give us a way to see which file created which count.
 2018-03-14 17:04:58 +00:00
Chris Hill-Scott
b878705caa Merge pull request #1769 from alphagov/escape-special-characters-in-search 
Escape special characters in search by recipient
 2018-03-14 15:44:56 +00:00
Leo Hemsted
5871dee606 use joinedload to only hit the database once per request 
also:

* only include active orgs
* write lots of tests
 2018-03-14 15:39:31 +00:00