Commit Graph

409 Commits

Author SHA1 Message Date
Ken Tsang
f9a7a78e55 Refactor code
- removed redundant dao method
- also handle letter jobs
2017-10-11 21:48:47 +01:00
Ken Tsang
c29fc8cfa4 Add celery task to check job finished
- celery task called after countdown of 60 minutes from start of job processing
2017-10-11 18:14:56 +01:00
Ken Tsang
dcf0d22d7b Added alert when job.notification_count doesn't match total notification for job
- Added log for when a job starts so that we will know when the processing of a job starts with the number of notifications
- Added dao method to get total notifications for a job id
- Added a test to check whether the number of notifications in the table matches the job notification_count
2017-10-10 15:04:55 +01:00
Imdad Ahad
d21d9cabd1 Use new default letter contact in the DVLA celery task 2017-10-03 10:31:15 +01:00
Leo Hemsted
f3db920c71 remove jobs from letter api calls
we now no longer create a job. At the end of the post there is no
action, as we don't have any tasks to queue immediately - if it's a
real notification it'll get picked up in the evening scheduled task.

If it's a test notification, we create it with an initial status of
sending so that we can be sure it'll never get picked up - and then we
trigger the update-letter-notifications-to-sent-to-dvla task to sent
the sent-at/by.
2017-09-26 09:57:36 +01:00
Leo Hemsted
17ba8db97f remove jobs from letter api and make success/error ftp callback tasks
1. No longer create jobs when creating letters from api 🎉
2. Bulk update notifications based on the notification references after
   we send them to DVLA - either as success or as error
2017-09-26 09:57:35 +01:00
Leo Hemsted
7dd3c1df5a set letter notifications to pending while notify-ftp does its stuff
this means that if the task is accidentally ran twice (eg we autoscale
notify-celery-worker-beat to 2), it won't send letters twice.

Additionally, update some function names and config variables to make
it clear that they are referring to letter jobs, rather than all letter
content
2017-09-26 09:57:35 +01:00
Leo Hemsted
f61ccd8ff0 add run_letter_notifications scheduled task
this task grabs all notifications that are sent via the API, and
are still in created - and sends them off to DVLA.
2017-09-26 09:57:35 +01:00
Rebecca Law
19f964a90b Added a check that the call is not using a test api key.
Removed the tests for trial mode service for the scheduled tasks and the process job.
Having the validation in the POST notification and create job endpoint is enough.
Updated the test_service_whitelist test because the order of the array is not gaurenteed.
2017-09-04 17:24:41 +01:00
Ken Tsang
d391919677 Refactored to check trial when running scheduled job 2017-08-30 22:30:05 +01:00
Ken Tsang
225c85832c Refactored to update job status and not build dvla file 2017-08-30 16:04:10 +01:00
Ken Tsang
01830b7e59 Push letter job to research queue in research mode 2017-08-30 16:04:10 +01:00
Ken Tsang
7e70b44113 Error in task when letter template and in trial mode 2017-08-30 16:04:10 +01:00
Leo Hemsted
614880f6d9 send to send-sms-tasks and send-email-tasks instead of send-tasks 2017-07-21 13:49:37 +01:00
Leo Hemsted
4d33040653 add separate send-sms and send-email queues
we're reading from those two queues as well as teh existing send queue,
however for now we don't send anything to them
2017-07-20 16:19:38 +01:00
Leo Hemsted
6c61a3fc2a Revert celery4
Revert the following three pull requests:
https://github.com/alphagov/notifications-api/pull/1085
https://github.com/alphagov/notifications-api/pull/1086
https://github.com/alphagov/notifications-api/pull/1088

celery 4.0.2 looked promising, however, on staging under mild load
(5/sec api calls) the performance was actually worse than 3.1.25
2017-07-19 15:17:19 +01:00
Martyn Inglis
786adb5d71 Move Queuenames in with the celery code, revamp config to allow move to celery 4.x 2017-07-12 12:01:52 +01:00
Rebecca Law
5ef0450e37 Update the json in the post to the service for the inbound sms. 2017-06-22 10:15:08 +01:00
Rebecca Law
6559da3239 Updates as per review comments 2017-06-21 15:29:55 +01:00
Rebecca Law
c608f5997b New task to send the inbound sms to the service.
If the service has not set the url then nothing happens.
If the request to the service url returns with 500 or greater the task is retries.
The task is created when the SMS provider post the inbound SMS.
2017-06-20 17:13:40 +01:00
Martyn Inglis
75c5e5e5d5 Merge branch 'master' into redo-queue-visibitlity-timeout
Conflicts:
	app/notifications/process_notifications.py
	app/v2/notifications/post_notifications.py
2017-05-30 10:18:18 +01:00
Imdad Ahad
cbc92a6173 Store the normalised number on the notification 2017-05-26 10:26:07 +01:00
Martyn Inglis
2591d3a1df This massive set of changes uses the new queue names object throughout the app and tests.
Lots of changes, all changing the line of code that puts things into queues, and the code that tests that.
2017-05-25 10:51:49 +01:00
Imdad Ahad
02db3be37c General refactor 2017-05-22 10:12:18 +01:00
Imdad Ahad
ee484ec368 Add get_s3_file method for use in DVLA processing 2017-05-15 10:48:37 +01:00
Imdad Ahad
7a10a91262 Revert "Process SNS request triggered by a DVLA S3 update" 2017-05-12 17:21:07 +01:00
Imdad Ahad
0f7093fc38 Refactor and add filename in logging 2017-05-12 14:32:01 +01:00
Imdad Ahad
95aa5680f9 Add more logging in update letter notifications task 2017-05-12 14:24:27 +01:00
Imdad Ahad
8a5e82904e Update to pull from correct bucket and fix tests not mocking out correctly 2017-05-12 14:24:26 +01:00
Imdad Ahad
f766f90207 Add task to process a DVLA response file:
* Currently we do nothing with the parsed response. We will
* update the status of the notifications in a separate PR
2017-05-12 14:24:26 +01:00
Imdad Ahad
c4fac1d937 Revert "Revert "add DVLA organisations to API"" 2017-04-21 16:05:07 +01:00
Leo Hemsted
c3e56d5d2d Revert "add DVLA organisations to API" 2017-04-20 18:21:56 +01:00
Chris Hill-Scott
5720f056ee Pass DVLA organisation ID through to Template
So that we know which organisation ID to spit out in the file.
2017-04-20 12:06:51 +01:00
minglis
ddda2bd158 Merge pull request #902 from alphagov/task-to-set-job-to-error
Task to set job to error
2017-04-18 16:07:32 +01:00
Martyn Inglis
8956338d31 Add a new task to update a job to error - note it leaves the notifications in whatever state they were in. 2017-04-18 11:42:48 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Rebecca Law
e022fe0403 Update build_dvla_file with a mocked notify_celery.send_task.
Check the task is being called with the right parameters.
2017-04-11 11:13:47 +01:00
Rebecca Law
6949779544 Refactor s3upload
s3upload function has been moved to notifications-utils.
https://github.com/alphagov/notifications-utils/pull/138 must be merge first.

Remove scratch code that I was using to test celery.send_task method

Revert the update to default_retry_delay.
Increase the countdown, wait 1 minute before executing the task.

Some of the changes in my previous commit are included here, that branch will be merged before this one, so it will make sense.

Update version of notification_utils

Update the version of notifications-utils
2017-04-10 17:17:58 +01:00
Rebecca Law
65a91c440b Refactor s3upload
s3upload function has been moved to notifications-utils.
https://github.com/alphagov/notifications-utils/pull/138 must be merge first.
2017-04-10 17:12:56 +01:00
Rebecca Law
2d9a449d1e Changed styling 2017-04-07 14:36:00 +01:00
Rebecca Law
50a5bedcbf Refactor update to notifcations to be a bulk update. This is much better for performance. 2017-04-07 10:59:12 +01:00
Rebecca Law
add0cfa62f Add a task for the FTP app to call that will update the job to sent to dvla and all the notifications for that job as sending. 2017-04-06 17:16:08 +01:00
Rebecca Law
6313640cbf Refactor if statement 2017-04-05 16:00:39 +01:00
Rebecca Law
c49217b1bf Update job status to ready to send, indicating the file for the job has been uploaded to S3. 2017-04-05 11:57:56 +01:00
Chris Hill-Scott
e47a008dd9 Pass through contact block from service to letter
Whatever a user has entered for their service’s contact block should
appear in the right place in the file we give to DVLA.

The work to output in the right fields in the DVLA file has already been
done. We just weren’t passing it through. This commit passes it through.
2017-04-03 13:27:47 +01:00
Chris Hill-Scott
c4c3268dd9 Use a simpler method of generating random number
Rather than generating each digit of the number, generate the whole
random number in one go. Should be faster, and is a bit easier to read.

Don’t need to worry about it not being zero-padded because the
`Template` class handles this here:
6ddd2ff352/notifications_utils/template.py (L410)
2017-04-03 13:25:22 +01:00
Chris Hill-Scott
7a412873d3 Refactor for better string concatenation
Using `+` to concatenate strings isn’t very memory efficient. Not sure
if there are real-world implications for how it’s being used here, but
can’t hurt to use `.join` instead.

Rewriting it as a generator also lets us remove some unneeded variable
assignment.
2017-04-03 13:23:33 +01:00
Martyn Inglis
7dcd3164e2 Revert "Reinstating the 2 task model for API submitted notifications."
This reverts commit 1c154c4113.
2017-03-30 10:46:23 +01:00
Martyn Inglis
9ee26e0b67 Revert "update notification rest tests"
This reverts commit 11919d9810.
2017-03-30 10:38:51 +01:00
Martyn Inglis
385a73da2b Revert "ensure we're passing through api keys and key types from notifications"
This reverts commit 25d1777937.
2017-03-30 10:38:41 +01:00