Commit Graph

326 Commits

Author SHA1 Message Date
Rebecca Law
5ef0450e37 Update the json in the post to the service for the inbound sms. 2017-06-22 10:15:08 +01:00
Rebecca Law
6559da3239 Updates as per review comments 2017-06-21 15:29:55 +01:00
Rebecca Law
c608f5997b New task to send the inbound sms to the service.
If the service has not set the url then nothing happens.
If the request to the service url returns with 500 or greater the task is retries.
The task is created when the SMS provider post the inbound SMS.
2017-06-20 17:13:40 +01:00
Imdad Ahad
f29ebb34f6 Add scheduled task to delete response files older than 7 days 2017-06-13 12:40:58 +01:00
Imdad Ahad
c4a89b77e3 Add task to remove transformed dvla files:
* Runs at 4:40am
* Remove files older than 7 days and only two day's worth
2017-06-07 16:39:23 +01:00
Imdad Ahad
74a8905be9 Seperate deletion of jobs:
* Two separate jobs, one for sms&email and another for letter
* Change celery task for delete to accept template type filter
* General refacor of tests to make more readable
2017-06-06 16:02:01 +01:00
Imdad Ahad
56c3f3cf7c Add task to delete inbound sms everyday at 1am 2017-06-02 16:10:27 +01:00
Martyn Inglis
75c5e5e5d5 Merge branch 'master' into redo-queue-visibitlity-timeout
Conflicts:
	app/notifications/process_notifications.py
	app/v2/notifications/post_notifications.py
2017-05-30 10:18:18 +01:00
Imdad Ahad
cbc92a6173 Store the normalised number on the notification 2017-05-26 10:26:07 +01:00
Martyn Inglis
4768f0b9fd Change retries policy.
Before we had a long back off, now we have more, but shorter backoffs.

- PREVIOUS
When we had an error talking to a provider we retried quickly and if we still got errors we backed off more and more. Maximum attempts was 5, max delay 4hours. This was to allow us time to ship a build if that was required.

- NOW
Backing off 48 times of 5 minutes each. This gives us the same total backoff, but many more tries in that period.

- WHY
Having the long back off meant messages could be delayed 4 hours. This was happening more and more, as PaaS deploys can place things into the "inflight" state in SQS. The inflight state MUST have an expiry time LONGER than the maximum retry back off. This meant that messages would be delayed 4 hours, even when there was no app error.

By doing this we can reduce this delay to 5 minutes. Whilst still giving us time to fix issues.
2017-05-25 11:12:40 +01:00
Martyn Inglis
2591d3a1df This massive set of changes uses the new queue names object throughout the app and tests.
Lots of changes, all changing the line of code that puts things into queues, and the code that tests that.
2017-05-25 10:51:49 +01:00
Rebecca Law
96b0e42959 Merge branch 'master' into schedule-api-notification
Conflicts:
	app/celery/scheduled_tasks.py
	tests/app/celery/test_scheduled_tasks.py
2017-05-24 13:21:22 +01:00
Martyn Inglis
a5c71ea907 New task to use the new DAO method to delete the data by type 2017-05-23 13:40:36 +01:00
Rebecca Law
76d0783c63 Add beat config for send-scheduled-notifications task to run every 15 minutes.
Added the missing commit to the update pending
2017-05-22 16:30:45 +01:00
Rebecca Law
9bfba52f53 Add pending flag to scheduled_notifications.
Set pending flag to false when the notification has been sent to provider task.
2017-05-22 15:07:16 +01:00
Rebecca Law
3a3161ecc4 Merge branch 'master' into schedule-api-notification
Conflicts:
	app/celery/scheduled_tasks.py
	app/v2/notifications/post_notifications.py
	tests/app/celery/test_scheduled_tasks.py
2017-05-22 14:05:57 +01:00
Imdad Ahad
02db3be37c General refactor 2017-05-22 10:12:18 +01:00
Rebecca Law
5f8338dd80 Remove the beat job.
Add logging to the task.
2017-05-17 12:59:00 +01:00
Martyn Inglis
83a1b1526e Merge branch 'master' into async-job-stats
Conflicts:
	app/notifications/notifications_ses_callback.py
2017-05-17 09:49:43 +01:00
Rebecca Law
2e078f9fc8 Add scheduled task to send scheduled notifcations.
Fix the query to fetch scheduled notifications.
2017-05-16 13:47:22 +01:00
Imdad Ahad
ee484ec368 Add get_s3_file method for use in DVLA processing 2017-05-15 10:48:37 +01:00
Imdad Ahad
7a10a91262 Revert "Process SNS request triggered by a DVLA S3 update" 2017-05-12 17:21:07 +01:00
Imdad Ahad
0f7093fc38 Refactor and add filename in logging 2017-05-12 14:32:01 +01:00
Imdad Ahad
95aa5680f9 Add more logging in update letter notifications task 2017-05-12 14:24:27 +01:00
Imdad Ahad
8a5e82904e Update to pull from correct bucket and fix tests not mocking out correctly 2017-05-12 14:24:26 +01:00
Imdad Ahad
f766f90207 Add task to process a DVLA response file:
* Currently we do nothing with the parsed response. We will
* update the status of the notifications in a separate PR
2017-05-12 14:24:26 +01:00
Martyn Inglis
eb07fce3ee Renamed method. 2017-05-12 12:19:56 +01:00
Martyn Inglis
8818bd5dba Scheduled task to call the timeout function 2017-05-11 15:22:57 +01:00
Martyn Inglis
76494a285e New logic to:
- increment sent only if the notifications is in a NON-finished state
- Increment outcome only if notification is in a finished state
2017-05-11 12:10:46 +01:00
Martyn Inglis
a4549a2e26 Put the statistics tasks into a new queue. This currently has no worker in PaaS so blocks the deployment until new PaaS worker is configured. 2017-05-09 22:03:13 +01:00
Martyn Inglis
f2a47044a4 Add a wrapper method around calling the tasks to manage when to call stats tasks 2017-05-09 18:17:26 +01:00
Martyn Inglis
b4bf332ea0 Added in tasks for created initial job stats and for updating following an outcome. 2017-05-09 15:24:05 +01:00
Martyn Inglis
2cbf8d29fd Removed white space for pep8 2017-05-09 14:46:15 +01:00
Martyn Inglis
b669c4930a Building the methods around async stats work 2017-05-09 12:30:31 +01:00
Imdad Ahad
b5d4acb758 Make message more accurate 2017-04-27 16:58:00 +01:00
Imdad Ahad
c4fac1d937 Revert "Revert "add DVLA organisations to API"" 2017-04-21 16:05:07 +01:00
Leo Hemsted
c3e56d5d2d Revert "add DVLA organisations to API" 2017-04-20 18:21:56 +01:00
Chris Hill-Scott
5720f056ee Pass DVLA organisation ID through to Template
So that we know which organisation ID to spit out in the file.
2017-04-20 12:06:51 +01:00
minglis
ddda2bd158 Merge pull request #902 from alphagov/task-to-set-job-to-error
Task to set job to error
2017-04-18 16:07:32 +01:00
Martyn Inglis
8956338d31 Add a new task to update a job to error - note it leaves the notifications in whatever state they were in. 2017-04-18 11:42:48 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Rebecca Law
e022fe0403 Update build_dvla_file with a mocked notify_celery.send_task.
Check the task is being called with the right parameters.
2017-04-11 11:13:47 +01:00
Rebecca Law
6949779544 Refactor s3upload
s3upload function has been moved to notifications-utils.
https://github.com/alphagov/notifications-utils/pull/138 must be merge first.

Remove scratch code that I was using to test celery.send_task method

Revert the update to default_retry_delay.
Increase the countdown, wait 1 minute before executing the task.

Some of the changes in my previous commit are included here, that branch will be merged before this one, so it will make sense.

Update version of notification_utils

Update the version of notifications-utils
2017-04-10 17:17:58 +01:00
Rebecca Law
65a91c440b Refactor s3upload
s3upload function has been moved to notifications-utils.
https://github.com/alphagov/notifications-utils/pull/138 must be merge first.
2017-04-10 17:12:56 +01:00
Rebecca Law
2d9a449d1e Changed styling 2017-04-07 14:36:00 +01:00
Rebecca Law
50a5bedcbf Refactor update to notifcations to be a bulk update. This is much better for performance. 2017-04-07 10:59:12 +01:00
Rebecca Law
add0cfa62f Add a task for the FTP app to call that will update the job to sent to dvla and all the notifications for that job as sending. 2017-04-06 17:16:08 +01:00
Rebecca Law
4f6a66c904 Merge branch 'master' into update-job-status
Conflicts:
	app/dao/jobs_dao.py
	tests/app/dao/test_jobs_dao.py
2017-04-06 11:11:37 +01:00
minglis
29c84d73ff Merge pull request #889 from alphagov/restrict-delete-files-job-query
Updates to the delete CSV file job to reduce the number jobs included in any run.
2017-04-06 10:09:49 +01:00
minglis
4d5e697f1c Merge pull request #888 from alphagov/throw-exception-on-long-running-client-call
Throw exception on long running client call
2017-04-06 09:24:26 +01:00