Commit Graph

262 Commits

Author SHA1 Message Date
Martyn Inglis
83a1b1526e Merge branch 'master' into async-job-stats
Conflicts:
	app/notifications/notifications_ses_callback.py
2017-05-17 09:49:43 +01:00
Imdad Ahad
4003edfa67 Add DVLA callback:
* Process SNS callback, trigger the update notifications celery task
* Put autoconfirm into its own method and use in callbacks
2017-05-15 11:12:31 +01:00
Imdad Ahad
ee484ec368 Add get_s3_file method for use in DVLA processing 2017-05-15 10:48:37 +01:00
Imdad Ahad
7a10a91262 Revert "Process SNS request triggered by a DVLA S3 update" 2017-05-12 17:21:07 +01:00
Imdad Ahad
0f7093fc38 Refactor and add filename in logging 2017-05-12 14:32:01 +01:00
Imdad Ahad
8a5e82904e Update to pull from correct bucket and fix tests not mocking out correctly 2017-05-12 14:24:26 +01:00
Imdad Ahad
f766f90207 Add task to process a DVLA response file:
* Currently we do nothing with the parsed response. We will
* update the status of the notifications in a separate PR
2017-05-12 14:24:26 +01:00
Martyn Inglis
1a61f06f53 Removed test as no longer make distinction on status. 2017-05-12 12:21:36 +01:00
Martyn Inglis
8818bd5dba Scheduled task to call the timeout function 2017-05-11 15:22:57 +01:00
Martyn Inglis
76494a285e New logic to:
- increment sent only if the notifications is in a NON-finished state
- Increment outcome only if notification is in a finished state
2017-05-11 12:10:46 +01:00
Martyn Inglis
a4549a2e26 Put the statistics tasks into a new queue. This currently has no worker in PaaS so blocks the deployment until new PaaS worker is configured. 2017-05-09 22:03:13 +01:00
Martyn Inglis
f2a47044a4 Add a wrapper method around calling the tasks to manage when to call stats tasks 2017-05-09 18:17:26 +01:00
Martyn Inglis
b4bf332ea0 Added in tasks for created initial job stats and for updating following an outcome. 2017-05-09 15:24:05 +01:00
Imdad Ahad
c4fac1d937 Revert "Revert "add DVLA organisations to API"" 2017-04-21 16:05:07 +01:00
Martyn Inglis
5550201939 Upper case only in the random notification id generator
as DVLA need upper case only or the barcode doesn't generate.
2017-04-20 10:55:53 +01:00
Leo Hemsted
935e566284 add tests for letters not getting excluded 2017-04-19 14:22:25 +01:00
minglis
ddda2bd158 Merge pull request #902 from alphagov/task-to-set-job-to-error
Task to set job to error
2017-04-18 16:07:32 +01:00
Martyn Inglis
2b4043515f fixed tests that used the new state. 2017-04-18 14:40:48 +01:00
Martyn Inglis
8956338d31 Add a new task to update a job to error - note it leaves the notifications in whatever state they were in. 2017-04-18 11:42:48 +01:00
Martyn Inglis
739ff2dceb Extra test on personilisation block 2017-04-18 11:07:11 +01:00
Martyn Inglis
1894c96f3a updated test to use random string not numeric id formats 2017-04-13 12:52:49 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Rebecca Law
e022fe0403 Update build_dvla_file with a mocked notify_celery.send_task.
Check the task is being called with the right parameters.
2017-04-11 11:13:47 +01:00
Rebecca Law
cf3eeb2120 Merge branch 'master' into task-to-update-letter-job
Conflicts:
	tests/app/celery/test_tasks.py
2017-04-06 17:18:09 +01:00
Rebecca Law
add0cfa62f Add a task for the FTP app to call that will update the job to sent to dvla and all the notifications for that job as sending. 2017-04-06 17:16:08 +01:00
Chris Hill-Scott
8bd386028b Merge pull request #893 from alphagov/dont-output-to_name_2
Build DVLA file with blank `TO_NAME_2` field
2017-04-06 16:54:02 +01:00
Chris Hill-Scott
7a6d748c6a Build DVLA file with blank TO_NAME_2 field
Problem: we were sending the first line of the address in the
`TO_NAME_2` field. This meant that they couldn’t do any PAF lookups with
it, because it wasn’t where they were expecting.

The first line of the address is the second line of what our users give
us. We need to give this to DVLA as the _third_ line of the address
output, which they call `TO_ADDRESS_LINE_1`.
2017-04-06 13:12:05 +01:00
Rebecca Law
4f6a66c904 Merge branch 'master' into update-job-status
Conflicts:
	app/dao/jobs_dao.py
	tests/app/dao/test_jobs_dao.py
2017-04-06 11:11:37 +01:00
Martyn Inglis
80965e4651 Good spot by Leo on has_calls not being exclusive,
Using call_args_list to get accurate list of args
2017-04-05 16:50:32 +01:00
Martyn Inglis
832005efef Updates to the delete CSV file job to reduce the number of eligible jobs in any run
- previously this was unbounded, so it got all jobs older then 7 days. In excess of 75,000 🔥
- this meant that the job took (a) a long time and (b) a lot memory and (c) doing the same thing every day

These changes mean that the job has a 2 day eligible window for jobs, minimising the number of eligible jobs in a run, whilst still retaining some leeway in event if it failing one night.

In principle the job runs early morning on a given day. The previous 7 days are left along, and then the previous 2 days worth of files are deleted:

so:
runs on
31st
30,29,28,27,26,25,24 are ignored
23,22 jobs here have files deleted
21 and earlier are ignored.
2017-04-05 16:23:41 +01:00
Chris Hill-Scott
61a8097ca5 Merge pull request #886 from alphagov/revert-883-revert-867-bump-utils-lists-as-placeholder-values
Revert "Revert "Bump utils to allow lists as placeholder values""
2017-04-05 13:14:57 +01:00
Chris Hill-Scott
f98f220e74 Update utils to fix letter output
Includes:
- [x] https://github.com/alphagov/notifications-utils/pull/136
- [ ] https://github.com/alphagov/notifications-utils/pull/131
2017-04-05 12:35:28 +01:00
Rebecca Law
c49217b1bf Update job status to ready to send, indicating the file for the job has been uploaded to S3. 2017-04-05 11:57:56 +01:00
Chris Hill-Scott
e47a008dd9 Pass through contact block from service to letter
Whatever a user has entered for their service’s contact block should
appear in the right place in the file we give to DVLA.

The work to output in the right fields in the DVLA file has already been
done. We just weren’t passing it through. This commit passes it through.
2017-04-03 13:27:47 +01:00
Chris Hill-Scott
eb5f9421dd Check for what DVLALetterTemplate is called with
Extends the test to make sure that the thing that builds each line of
the file is getting called with the right template, personalisation and
numeric ID. Will be helpful the more complicated the call to the
template gets.
2017-04-03 13:25:22 +01:00
Martyn Inglis
7dcd3164e2 Revert "Reinstating the 2 task model for API submitted notifications."
This reverts commit 1c154c4113.
2017-03-30 10:46:23 +01:00
Martyn Inglis
385a73da2b Revert "ensure we're passing through api keys and key types from notifications"
This reverts commit 25d1777937.
2017-03-30 10:38:41 +01:00
Leo Hemsted
25d1777937 ensure we're passing through api keys and key types from notifications
when we made the change to async persist notifications, we forgot to
pass through api_key_id and key_type. in send_sms/email, for legacy
reasons, they default to None/KEY_TYPE_NORMAL, so regardless of what
your api key was set up as, we would send real messages!

TODO: Once the PaaS transition is complete and the task changes are
reverted, remove the api_key_id and key_type params from the send_*
tasks entirely, as those are only called from the csv job flow, and
don't need them
2017-03-28 13:14:46 +01:00
Rebecca Law
31979c3272 Use 500 for org id 2017-03-27 17:49:09 +01:00
minglis
d7adfcaf87 Merge pull request #866 from alphagov/no-downtime-hosting-migration
Reinstating the 2 task model for API submitted notifications.
2017-03-27 13:54:43 +01:00
Rebecca Law
37a5ce601a Update the version of notifications-utils.
There was a problem with the address block for the DVLATemplate.

Added a countdown to the build_dvla_file.
2017-03-24 15:25:02 +00:00
Martyn Inglis
1c154c4113 Reinstating the 2 task model for API submitted notifications.
This is being done for the PaaS migration to allow us to keep traffic coming in whilst we migrate the database.

uses the same tasks as the CSV uploaded notifications. Simple changes to not persist the notification, and call into a different task.
2017-03-23 14:41:00 +00:00
Rebecca Law
140179b4b6 Create new task to build dvla file.
This will transform each notification in a job to a row in a file.
The file is then uploaded to S3.
The files will later be aggregated by the notifications-ftp app to send to dvla.

The method to upload the file to S3 should be pulled into notifications-utils package.
It is the same method used in notifications-admin.
2017-03-15 15:26:58 +00:00
imdadahad
c11f4b5d5d Merge pull request #829 from alphagov/feat-add-endpoint-for-service-monthly-usage-stats
Add endpoint for service monthly usage stats by template
2017-03-09 16:12:22 +00:00
Imdad Ahad
a4dc614ef7 Restore dao_create_template and use custom template fixture instead 2017-03-08 13:03:44 +00:00
Imdad Ahad
8272a4388d Add dao method to get template stats per month in financial year for service 2017-03-08 12:01:38 +00:00
Imdad Ahad
37341e7a62 Updates:
* Add notify user id in config
* Add dao method to get provider history versions along with tests
* BUG: Provider switching did not handle case where priorities were equal. This
* adds a fix to properly cover this case along with tests
2017-03-03 10:54:20 +00:00
Imdad Ahad
d805985a4e Refactor tests to use cleaner fixture and be more verbose in tests 2017-02-27 13:18:42 +00:00
Imdad Ahad
204d72830f Update switch task to use sent_at and newer db helpers 2017-02-24 13:41:32 +00:00
Imdad Ahad
73d5ce4f8b Add tests to verify correctness of the switching provider task 2017-02-24 12:23:39 +00:00