Commit Graph

311 Commits

Author SHA1 Message Date
Katie Smith
0c9a4bce59 Add celery tasks- save_sms, save_email, save_letter
Created three new celery tasks:
* save_sms          (will replace send_sms)
* save_email        (will replace send_email)
* save_letter       (will replace persist_letter)

The difference between the new tasks and the tasks they are replacing is
that we no longer pass in the datetime as a parameter.

The code has been changed to use the new tasks, and the tests now run
against the new tasks too. The old tasks will need be removed in a separate
commit.
2017-10-18 17:00:37 +01:00
Rebecca Law
24a87d8260 Small change to use the current datetime stamp for the Notification.created_at timestamp rather than the job.created_at start date. This will give us better more reflective data. 2017-10-17 16:05:31 +01:00
Richard Chapman
543535e2af Process Incomplete Jobs Updates
Comments are PR review. Updated code style in a few places to make it
more consistent with other code, added tests for letters and emails
so they are testedt, refactored some database queries to dao file

- Fixed code style
- Refactored database queries to dao code
- Added tests for emails and sms.
2017-10-17 11:07:36 +01:00
Richard Chapman
0ff80bcb76 Process Incomplete Jobs Refactor
- Moved the process_incomplete_jobs to tasks.py
- Moved the process_incomplete_jobs test to test_tasks.py
- Cleaned up imports and other code style issues.

As the new tasks is not a scheduled one, moved the the tasks.py file.
This makes it more consisted with other tasks. Updated a few code style
issues to make it more consistent with other coe and hence more
maintainable in future.
2017-10-16 12:33:39 +01:00
Rebecca Law
e08690cad2 Added a new scheduled task that runs every minute, may want to run it every 3 minutes.
The tasks checks that a job is not still in progress 30 minutes after the job started processing.
2017-10-12 16:21:08 +01:00
Ken Tsang
f9a7a78e55 Refactor code
- removed redundant dao method
- also handle letter jobs
2017-10-11 21:48:47 +01:00
Ken Tsang
c29fc8cfa4 Add celery task to check job finished
- celery task called after countdown of 60 minutes from start of job processing
2017-10-11 18:14:56 +01:00
Ken Tsang
dcf0d22d7b Added alert when job.notification_count doesn't match total notification for job
- Added log for when a job starts so that we will know when the processing of a job starts with the number of notifications
- Added dao method to get total notifications for a job id
- Added a test to check whether the number of notifications in the table matches the job notification_count
2017-10-10 15:04:55 +01:00
Imdad Ahad
d21d9cabd1 Use new default letter contact in the DVLA celery task 2017-10-03 10:31:15 +01:00
Leo Hemsted
c97b130540 test dvla callback update tasks 2017-09-26 12:36:42 +01:00
Leo Hemsted
7dd3c1df5a set letter notifications to pending while notify-ftp does its stuff
this means that if the task is accidentally ran twice (eg we autoscale
notify-celery-worker-beat to 2), it won't send letters twice.

Additionally, update some function names and config variables to make
it clear that they are referring to letter jobs, rather than all letter
content
2017-09-26 09:57:35 +01:00
Leo Hemsted
f61ccd8ff0 add run_letter_notifications scheduled task
this task grabs all notifications that are sent via the API, and
are still in created - and sends them off to DVLA.
2017-09-26 09:57:35 +01:00
Leo Hemsted
e18e78180e update letter tests to use correct service
previously they were using sample_service fixture under the hood, but
with full permissions added - this works fine, **unless** there's
already a service with the name "sample service" in the database. This
can happen for two reasons:

* A previous test didn't tear down correctly
* This test already invoked the sample_service fixture somehow

If this happens, we just return the existing service, without modifying
its values - values that we might change in tests, such as
research mode or letters permissions.

In the future, we'll have to be vigilant! and aware! and careful! to
not use sample_service if we're doing tests involving letters, since
they create a service with a different name now
2017-09-21 11:50:49 +01:00
Katie Smith
dbeec6c88b Remove unused fixtures from tests 2017-09-05 11:50:39 +01:00
Rebecca Law
19f964a90b Added a check that the call is not using a test api key.
Removed the tests for trial mode service for the scheduled tasks and the process job.
Having the validation in the POST notification and create job endpoint is enough.
Updated the test_service_whitelist test because the order of the array is not gaurenteed.
2017-09-04 17:24:41 +01:00
Ken Tsang
225c85832c Refactored to update job status and not build dvla file 2017-08-30 16:04:10 +01:00
Ken Tsang
01830b7e59 Push letter job to research queue in research mode 2017-08-30 16:04:10 +01:00
Ken Tsang
7e70b44113 Error in task when letter template and in trial mode 2017-08-30 16:04:10 +01:00
Leo Hemsted
c2152f9cb8 remove billable-units endpoint
it wasn't used anywhere, and the return value changed when flask was upgraded
2017-08-21 10:57:15 +01:00
Leo Hemsted
c36e50bef1 update dependencies 2017-08-18 17:02:31 +01:00
Leo Hemsted
614880f6d9 send to send-sms-tasks and send-email-tasks instead of send-tasks 2017-07-21 13:49:37 +01:00
Rebecca Law
5ef0450e37 Update the json in the post to the service for the inbound sms. 2017-06-22 10:15:08 +01:00
Rebecca Law
c608f5997b New task to send the inbound sms to the service.
If the service has not set the url then nothing happens.
If the request to the service url returns with 500 or greater the task is retries.
The task is created when the SMS provider post the inbound SMS.
2017-06-20 17:13:40 +01:00
Martyn Inglis
2591d3a1df This massive set of changes uses the new queue names object throughout the app and tests.
Lots of changes, all changing the line of code that puts things into queues, and the code that tests that.
2017-05-25 10:51:49 +01:00
Imdad Ahad
02db3be37c General refactor 2017-05-22 10:12:18 +01:00
Imdad Ahad
4003edfa67 Add DVLA callback:
* Process SNS callback, trigger the update notifications celery task
* Put autoconfirm into its own method and use in callbacks
2017-05-15 11:12:31 +01:00
Imdad Ahad
ee484ec368 Add get_s3_file method for use in DVLA processing 2017-05-15 10:48:37 +01:00
Imdad Ahad
7a10a91262 Revert "Process SNS request triggered by a DVLA S3 update" 2017-05-12 17:21:07 +01:00
Imdad Ahad
0f7093fc38 Refactor and add filename in logging 2017-05-12 14:32:01 +01:00
Imdad Ahad
8a5e82904e Update to pull from correct bucket and fix tests not mocking out correctly 2017-05-12 14:24:26 +01:00
Imdad Ahad
f766f90207 Add task to process a DVLA response file:
* Currently we do nothing with the parsed response. We will
* update the status of the notifications in a separate PR
2017-05-12 14:24:26 +01:00
Imdad Ahad
c4fac1d937 Revert "Revert "add DVLA organisations to API"" 2017-04-21 16:05:07 +01:00
Martyn Inglis
5550201939 Upper case only in the random notification id generator
as DVLA need upper case only or the barcode doesn't generate.
2017-04-20 10:55:53 +01:00
minglis
ddda2bd158 Merge pull request #902 from alphagov/task-to-set-job-to-error
Task to set job to error
2017-04-18 16:07:32 +01:00
Martyn Inglis
2b4043515f fixed tests that used the new state. 2017-04-18 14:40:48 +01:00
Martyn Inglis
8956338d31 Add a new task to update a job to error - note it leaves the notifications in whatever state they were in. 2017-04-18 11:42:48 +01:00
Martyn Inglis
739ff2dceb Extra test on personilisation block 2017-04-18 11:07:11 +01:00
Martyn Inglis
1894c96f3a updated test to use random string not numeric id formats 2017-04-13 12:52:49 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Rebecca Law
e022fe0403 Update build_dvla_file with a mocked notify_celery.send_task.
Check the task is being called with the right parameters.
2017-04-11 11:13:47 +01:00
Rebecca Law
cf3eeb2120 Merge branch 'master' into task-to-update-letter-job
Conflicts:
	tests/app/celery/test_tasks.py
2017-04-06 17:18:09 +01:00
Rebecca Law
add0cfa62f Add a task for the FTP app to call that will update the job to sent to dvla and all the notifications for that job as sending. 2017-04-06 17:16:08 +01:00
Chris Hill-Scott
8bd386028b Merge pull request #893 from alphagov/dont-output-to_name_2
Build DVLA file with blank `TO_NAME_2` field
2017-04-06 16:54:02 +01:00
Chris Hill-Scott
7a6d748c6a Build DVLA file with blank TO_NAME_2 field
Problem: we were sending the first line of the address in the
`TO_NAME_2` field. This meant that they couldn’t do any PAF lookups with
it, because it wasn’t where they were expecting.

The first line of the address is the second line of what our users give
us. We need to give this to DVLA as the _third_ line of the address
output, which they call `TO_ADDRESS_LINE_1`.
2017-04-06 13:12:05 +01:00
Rebecca Law
4f6a66c904 Merge branch 'master' into update-job-status
Conflicts:
	app/dao/jobs_dao.py
	tests/app/dao/test_jobs_dao.py
2017-04-06 11:11:37 +01:00
Chris Hill-Scott
61a8097ca5 Merge pull request #886 from alphagov/revert-883-revert-867-bump-utils-lists-as-placeholder-values
Revert "Revert "Bump utils to allow lists as placeholder values""
2017-04-05 13:14:57 +01:00
Chris Hill-Scott
f98f220e74 Update utils to fix letter output
Includes:
- [x] https://github.com/alphagov/notifications-utils/pull/136
- [ ] https://github.com/alphagov/notifications-utils/pull/131
2017-04-05 12:35:28 +01:00
Rebecca Law
c49217b1bf Update job status to ready to send, indicating the file for the job has been uploaded to S3. 2017-04-05 11:57:56 +01:00
Chris Hill-Scott
e47a008dd9 Pass through contact block from service to letter
Whatever a user has entered for their service’s contact block should
appear in the right place in the file we give to DVLA.

The work to output in the right fields in the DVLA file has already been
done. We just weren’t passing it through. This commit passes it through.
2017-04-03 13:27:47 +01:00
Chris Hill-Scott
eb5f9421dd Check for what DVLALetterTemplate is called with
Extends the test to make sure that the thing that builds each line of
the file is getting called with the right template, personalisation and
numeric ID. Will be helpful the more complicated the call to the
template gets.
2017-04-03 13:25:22 +01:00