darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2026-01-18 16:41:56 -05:00
Code Issues Packages Projects Releases Wiki Activity
4,974 Commits 134 Branches 60 Tags
c75458cee9d5d6782af12b769aeff1cfb3af276f
Commit Graph

4974 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rebecca Law
c75458cee9 Revert change to exception log. 2018-03-26 16:44:29 +01:00
Rebecca Law
598539dcb3 Update logging for provider tasks. 
Move the info message before the fetch.
Include the exception in the log message.
 2018-03-26 15:24:21 +01:00
Rebecca Law
bdae669b05 Merge pull request #1800 from alphagov/replay-created-emails 
Replay emails and sms that are still in created
 2018-03-26 10:56:55 +01:00
Rebecca Law
acfb1305f4 Fix test 2018-03-26 10:51:38 +01:00
Rebecca Law
0701b2546d Remove test crontab minute 2018-03-26 10:30:08 +01:00
Rebecca Law
9549ada200 Run task every 15 minutes. 
Move variable to task from config.
 2018-03-26 10:26:24 +01:00
Sakis
2ee48b5d4c Merge pull request #1799 from alphagov/python-3.5 
Update Python version to match PaaS
 2018-03-26 10:22:13 +01:00
Rebecca Law
612843d509 Run every 15 minutes not 15 minutes past the hour 2018-03-26 09:43:53 +01:00
Rebecca Law
28e78780d0 Added more logging for provider tasks. 2018-03-26 09:31:52 +01:00
Rebecca Law
b977860a4e Merge pull request #1801 from alphagov/command-to-create-pdf-letter 
New command to create-pdf-letter task for a given notification id.
 2018-03-23 16:20:04 +00:00
Rebecca Law
fc21121764 Check input is a UUID 2018-03-23 16:11:43 +00:00
Rebecca Law
40e535e112 Add the scheduled task to run every 15 minutes. 2018-03-23 16:00:13 +00:00
Rebecca Law
6dad9b43db New command to create-pdf-letter task for a given notification id. 
After a notificaiton is created we create a task to create the pdf and save it to S3,
if for some reason that task does not run we are left with notifications that are not sent.
This should not happen, but if it does we have a way to continue sending the letter.
 2018-03-23 15:47:01 +00:00
Rebecca Law
f596d17bf2 If a sms or email has not been sent after 4 hours and 15 minutes then put it on the delivery queue. 2018-03-23 15:38:35 +00:00
Chris Hill-Scott
d28aa9080a Update Python version to match PaaS 2018-03-23 15:36:50 +00:00
kentsanggds
5d38989cd9 Merge pull request #1797 from alphagov/ken-test-key-antivirus 
Add antivirus check on precompiled letters sent with test key
 2018-03-23 15:04:08 +00:00
Ken Tsang
1a9bc2a5cf Move test letters to test letters bucket without date folder name 2018-03-23 14:59:48 +00:00
Ken Tsang
2f09c943f7 Change error log to exception log for pdfs failing virus scans 2018-03-23 14:24:23 +00:00
Venus Bailey
d14ac2aa81 Merge pull request #1795 from alphagov/vb-report-tasks 
Fixed a bug on reporting_tasks
 2018-03-23 14:04:12 +00:00
Ken Tsang
0ee5c33084 Add antivirus check on precompiled letters sent with test key 
- precompiled PDFs sent by test key uploaded to scan bucket
- set status to VIRUS-SCAN-FAILED for pdfs failing virus scan rather than PERMANENT-FAILURE
- Make call to AV app for precompiled letters sent via a test key, and set notification status to PENDING-VIRUS-SCAN
 2018-03-23 12:04:37 +00:00
venusbb
55a27fed13 fixed a bug on reporting_task 2018-03-23 11:40:27 +00:00
Chris Waszczuk
9bf739007a Merge pull request #1790 from alphagov/update-paginated-inbound-sms-method 
Update inbound sms method
 2018-03-23 11:25:47 +00:00
chrisw
ac50250ad7 update paginated inbound sms method 2018-03-23 11:09:51 +00:00
kentsanggds
6e9bab5544 Merge pull request #1778 from alphagov/pyup-update-notifications-python-client-4.7.2-to-4.8.1 
Update notifications-python-client to 4.8.1
 2018-03-23 10:35:16 +00:00
kentsanggds
77de45b105 Merge pull request #1792 from alphagov/pyup-update-moto-1.2-to-1.3.1 
Update moto to 1.3.1
 2018-03-23 10:34:52 +00:00
kentsanggds
d793a55b9c Merge pull request #1779 from alphagov/pyup-update-pyjwt-1.6.0-to-1.6.1 
Update pyjwt to 1.6.1
 2018-03-23 10:34:09 +00:00
kentsanggds
832c350b69 Merge pull request #1789 from alphagov/ken-fix-no-sent_by 
Set sent_by if not set during sms provider callback
 2018-03-23 09:52:26 +00:00
pyup-bot
3acaba0a33 Update moto from 1.2 to 1.3.1 2018-03-22 18:40:55 +00:00
Venus Bailey
8c282aa406 Merge pull request #1791 from alphagov/vb-report-tasks 
Add statd to monitoring the length of task
 2018-03-22 17:22:36 +00:00
venusbb
444365faa5 add statd 2018-03-22 17:17:03 +00:00
Ken Tsang
c272d023c4 Lowercase the client name when sent_by is empty 2018-03-22 16:12:33 +00:00
Venus Bailey
1d7381400d Merge pull request #1787 from alphagov/vb-report-tasks 
Command for migrating data to ft_billing [#155511547]
 2018-03-22 09:37:46 +00:00
Ken Tsang
ffa093d8c7 Set sent_by if not set during sms provider callback 
- refactored argument reference to provider_reference to make it clearer
 2018-03-21 18:11:10 +00:00
venusbb
9aa2536997 use sql parametrize rather than python format 2018-03-21 17:04:51 +00:00
kentsanggds
5dc0248043 Merge pull request #1783 from alphagov/ken-process-antivirus 
Send task to antivirus app and process antivirus callbacks
 2018-03-21 16:39:55 +00:00
Ken Tsang
d6f6669610 Add comments for copying s3 objects 2018-03-21 16:33:49 +00:00
venusbb
4b25654cbf update record rather than ignore when duplicate 2018-03-21 15:37:49 +00:00
Ken Tsang
5481d743ac change failed virus scan log to error 2018-03-21 15:30:58 +00:00
venusbb
b9953dd005 Command to migrate data to ft_billing 2018-03-21 15:21:16 +00:00
Venus Bailey
40a3986720 Merge pull request #1786 from alphagov/vb-report-tasks 
Primary key for ft_billing table should use notification_type
 2018-03-21 15:18:23 +00:00
venusbb
af975b89ec primary key for ft_billing should use notification_type 2018-03-21 15:02:08 +00:00
Venus Bailey
b59a59f45a Merge pull request #1785 from alphagov/vb-report-tasks 
Add import to reporting_tasks in scheduled tasks
 2018-03-21 10:48:01 +00:00
venusbb
378feda603 put import reporting_tasks in config 2018-03-21 10:39:00 +00:00
venusbb
cf019864e2 Preserve 'unknown' in ft_billing for sms if the provider is not known. 2018-03-21 09:50:34 +00:00
venusbb
ecef20523b Merge branch 'master' of https://github.com/alphagov/notifications-api into vb-report-tasks 2018-03-20 17:24:32 +00:00
venusbb
07aa44ea82 Add import report_tasks to schedule_tasks. 2018-03-20 16:42:24 +00:00
Rebecca Law
81f9c06e06 Merge pull request #1781 from alphagov/only-use-data-on-queue-not-from-db 
Remove unused code in send_delivery_status_to_service task
 2018-03-20 16:13:42 +00:00
Rebecca Law
30cbc45872 Merge pull request #1784 from alphagov/optimize-todays-services-query 
Optimise query for platform-admin/live-services and trial-services
 2018-03-20 16:13:27 +00:00
Rebecca Law
0798154fa2 Optimize the query used to return the services and todays notification totals. 
By changing the created_at filter to use a specific date range I found a significant improvement to the queries performance.
The unit test needed to change because now were are returning todays date as BST the local timezone.

Query plan before

Merge Left Join  (cost=1226133.76..1226143.77 rows=1753 width=70) (actual time=5800.160..5801.657 rows=1849 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.205..2.631 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.011..1.156 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=1225980.77..1225980.99 rows=86 width=40) (actual time=5797.949..5797.984 rows=198 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=1225976.29..1225978.01 rows=86 width=40) (actual time=5797.682..5797.823 rows=198 loops=1)
->  HashAggregate  (cost=1225976.29..1225977.15 rows=86 width=48) (actual time=5797.681..5797.747 rows=198 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Seq Scan on notifications  (cost=0.00..1220610.86 rows=536543 width=48) (actual time=0.064..5482.975 rows=643799 loops=1)
Filter: (((key_type)::text <> 'TEST'::text) AND (date(created_at) = '2018-03-20'::date))
Rows Removed by Filter: 6804774
Planning time: 1.106 ms
Execution time: 5802.130 ms

Query plan after
Merge Left Join  (cost=953378.30..953388.30 rows=1753 width=70) (actual time=2380.144..2382.499 rows=1852 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.944..3.570 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.006..1.294 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=953225.31..953225.53 rows=86 width=40) (actual time=2377.194..2377.262 rows=201 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=953220.83..953222.55 rows=86 width=40) (actual time=2376.797..2377.034 rows=201 loops=1)
->  HashAggregate  (cost=953220.83..953221.69 rows=86 width=48) (actual time=2376.795..2376.905 rows=201 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Bitmap Heap Scan on notifications  (cost=29883.14..947856.24 rows=536459 width=48) (actual time=270.061..1887.754 rows=644735 loops=1)
Recheck Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Rows Removed by Index Recheck: 947427
Filter: ((key_type)::text <> 'TEST'::text)
Heap Blocks: exact=40882 lossy=186483
->  Bitmap Index Scan on ix_notifications_created_at  (cost=0.00..29749.02 rows=536459 width=0) (actual time=258.631..258.631 rows=644849 loops=1)
Index Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Planning time: 0.548 ms
Execution time: 2383.485 ms
 2018-03-20 15:48:32 +00:00
Ken Tsang
8391353113 Update tasks in line with AV app 2018-03-20 14:56:42 +00:00