Commit Graph

329 Commits

Author SHA1 Message Date
kentsanggds
c110ca3f57 Merge pull request #1303 from alphagov/ken-purge-noti-to-email_reply_to
Purge notifications to email reply to rows older than 7 days
2017-10-13 15:23:58 +01:00
Rebecca Law
e08690cad2 Added a new scheduled task that runs every minute, may want to run it every 3 minutes.
The tasks checks that a job is not still in progress 30 minutes after the job started processing.
2017-10-12 16:21:08 +01:00
Ken Tsang
f9a7a78e55 Refactor code
- removed redundant dao method
- also handle letter jobs
2017-10-11 21:48:47 +01:00
Ken Tsang
c29fc8cfa4 Add celery task to check job finished
- celery task called after countdown of 60 minutes from start of job processing
2017-10-11 18:14:56 +01:00
Ken Tsang
1eaf6e089b Refactor code to make it clearer
- Added code in `delete_notifications_created_more_than_a_week_ago_by_type` to remove notifications to email_reply_to older than 7 days
- Added transactional to `delete_notifications_created_more_than_a_week_ago_by_type`
2017-10-11 12:26:58 +01:00
Ken Tsang
dcf0d22d7b Added alert when job.notification_count doesn't match total notification for job
- Added log for when a job starts so that we will know when the processing of a job starts with the number of notifications
- Added dao method to get total notifications for a job id
- Added a test to check whether the number of notifications in the table matches the job notification_count
2017-10-10 15:04:55 +01:00
Ken Tsang
2c46f201b5 Purge noti to email reply to older than 7 days 2017-10-09 14:51:27 +01:00
Rebecca Law
78a9d74c49 Merge branch 'master' into return-service_sms_sender_value 2017-10-04 13:29:05 +01:00
Imdad Ahad
d21d9cabd1 Use new default letter contact in the DVLA celery task 2017-10-03 10:31:15 +01:00
Rebecca Law
0db39bfac8 Fix the unit tests.
I think there was some imports missed when resolving merge conflicts.
Also I'm not sure why the test_update_letter_notification_to_sent or error passed, I've updated them so they do pass.
2017-10-02 15:29:13 +01:00
Leo Hemsted
c97b130540 test dvla callback update tasks 2017-09-26 12:36:42 +01:00
Leo Hemsted
aaadf09562 add tests for new sched task 2017-09-26 12:03:06 +01:00
Leo Hemsted
7dd3c1df5a set letter notifications to pending while notify-ftp does its stuff
this means that if the task is accidentally ran twice (eg we autoscale
notify-celery-worker-beat to 2), it won't send letters twice.

Additionally, update some function names and config variables to make
it clear that they are referring to letter jobs, rather than all letter
content
2017-09-26 09:57:35 +01:00
Leo Hemsted
f61ccd8ff0 add run_letter_notifications scheduled task
this task grabs all notifications that are sent via the API, and
are still in created - and sends them off to DVLA.
2017-09-26 09:57:35 +01:00
Leo Hemsted
e18e78180e update letter tests to use correct service
previously they were using sample_service fixture under the hood, but
with full permissions added - this works fine, **unless** there's
already a service with the name "sample service" in the database. This
can happen for two reasons:

* A previous test didn't tear down correctly
* This test already invoked the sample_service fixture somehow

If this happens, we just return the existing service, without modifying
its values - values that we might change in tests, such as
research mode or letters permissions.

In the future, we'll have to be vigilant! and aware! and careful! to
not use sample_service if we're doing tests involving letters, since
they create a service with a different name now
2017-09-21 11:50:49 +01:00
Leo Hemsted
2c758cbccd use new send-jobs-to-dvla task instead of send-files-to-dvla
the tasks (on the ftp app) do the same thing, but to improve clarity
i've renamed it to jobs, because we'll be adding a notifications one
soon
2017-09-12 17:28:07 +01:00
Rebecca Law
e32abb0ada Merge pull request #1221 from alphagov/ken-no-letters-when-trial
Stop letters when service is in trial
2017-09-05 15:33:50 +01:00
Katie Smith
dbeec6c88b Remove unused fixtures from tests 2017-09-05 11:50:39 +01:00
Rebecca Law
19f964a90b Added a check that the call is not using a test api key.
Removed the tests for trial mode service for the scheduled tasks and the process job.
Having the validation in the POST notification and create job endpoint is enough.
Updated the test_service_whitelist test because the order of the array is not gaurenteed.
2017-09-04 17:24:41 +01:00
Ken Tsang
d391919677 Refactored to check trial when running scheduled job 2017-08-30 22:30:05 +01:00
Ken Tsang
225c85832c Refactored to update job status and not build dvla file 2017-08-30 16:04:10 +01:00
Ken Tsang
01830b7e59 Push letter job to research queue in research mode 2017-08-30 16:04:10 +01:00
Ken Tsang
7e70b44113 Error in task when letter template and in trial mode 2017-08-30 16:04:10 +01:00
Ken Tsang
41a71c703b Refactor test 2017-08-30 16:04:10 +01:00
Katie Smith
a1a5fdedb1 Send results of processing-time query to performance platform 2017-08-30 14:40:10 +01:00
Leo Hemsted
cf2d803db1 Merge pull request #1205 from alphagov/perf-stats
Perf stats refactor
2017-08-29 10:44:09 +01:00
Leo Hemsted
e85b621cbc make perf platform client handle more stuff sensibly
specifically, all of the performance platform specific data layout now
happens in performance_platform_client.py - stuff like setting the
_timestamp, period etc, and the perf platform-specific nomenclature is
all handled there.
2017-08-24 17:10:42 +01:00
Ken Tsang
1b35731fb2 Refactor code
- updated dao_get_letter_jobs_by_status to dao_get_letter_job_ids_by_status to return array of strings
2017-08-24 11:57:46 +01:00
Leo Hemsted
412c87cfc8 pycodestyle 2017-08-24 10:52:47 +01:00
Leo Hemsted
bd2682b521 add new performance-platform section to cf config
it's a new cf-service we've got to create, that contains endpoints
and the bearer tokens for them.
2017-08-23 18:14:35 +01:00
Ken Tsang
451c5a9f1a Made celery task arg a tuple 2017-08-23 18:05:48 +01:00
Leo Hemsted
89f4f5173e refactor performance platform code
so that it doesn't appear generic when it's actually specific to
sending the daily notification totals. To do this, split it out into a
separate performance_platform directory, containing the business logic,
and make the performance_platform_client incredibly thin - all it
handles is adding ids to payloads, and sending stats.

Also, some changes to the config (not all done yet) since there is one
token per endpoint, not one for the whole platform as we'd previously
coded
2017-08-23 17:37:29 +01:00
Ken Tsang
4fb5e68ce7 Refactor code
- Created TaskNames for DVLA_FILES rather than have DVLA_FILES in QueueNames
- Removed PROCESS_FTP from all_queues() as this was causing problems in picking up letter job tasks
- Created test to ensure that we don't arbitrarily add queue names to all_queues
2017-08-23 13:22:07 +01:00
Ken Tsang
b2b74e2b5d Fixed bug in letter job schedule 2017-08-23 13:22:07 +01:00
Ken Tsang
18881cd580 Add scheduled letter jobs 2017-08-23 13:22:07 +01:00
Leo Hemsted
c2152f9cb8 remove billable-units endpoint
it wasn't used anywhere, and the return value changed when flask was upgraded
2017-08-21 10:57:15 +01:00
Leo Hemsted
c36e50bef1 update dependencies 2017-08-18 17:02:31 +01:00
Imdad Ahad
94605d31fa Change how we populate and retrieve MonthlyBilling totals:
1. For both email and sms, store [] in monthly_totals if
there is no billing data (no notifications sent etc.) and
return this via the API

2. General refactoring of indentation
2017-08-15 16:09:42 +01:00
Imdad Ahad
63e1167098 Make sure we update the correct month for billing:
When populating the monthly billing records on a schedule, we need
to ensure the correct month is being updated.

As an example, if the current datetime is 31 Mar 2016, 23:00. The
BST equivalent is the 1st April. Therefore we need to ensure we
update billing for April, not March. This takes care of that.
2017-08-11 16:57:17 +01:00
Rebecca Law
c1f2634c90 Removed month and year and replaced it with start_date and end_date.
This will allow us to sort the data properly.
2017-07-26 13:19:17 +01:00
Rebecca Law
91f29517eb Merge branch 'master' into month-billing-table 2017-07-24 15:17:36 +01:00
Rebecca Law
3e2b8190b9 - Added a scheduled task to create or update billing for the month, yesterday is used to calculate the start and end date for the month.
- The new task has not been added to the beat application yet.
- Added an updated_at column to the monthly billing table, we may want to only calculate from the last updated date rather than the entire month.
2017-07-24 15:13:18 +01:00
Leo Hemsted
614880f6d9 send to send-sms-tasks and send-email-tasks instead of send-tasks 2017-07-21 13:49:37 +01:00
Leo Hemsted
6c61a3fc2a Revert celery4
Revert the following three pull requests:
https://github.com/alphagov/notifications-api/pull/1085
https://github.com/alphagov/notifications-api/pull/1086
https://github.com/alphagov/notifications-api/pull/1088

celery 4.0.2 looked promising, however, on staging under mild load
(5/sec api calls) the performance was actually worse than 3.1.25
2017-07-19 15:17:19 +01:00
Martyn Inglis
0e9e8955f7 Finished celery refactor - set up config for queue prefix
LEO notes: Also made sure the Test BROKER_URL is preserved so that
tests warn you when celery isn't mocked out
2017-07-12 12:37:18 +01:00
Imdad Ahad
c8c47f44a9 Don't update older status in timeout notifications task 2017-07-10 14:09:30 +01:00
Imdad Ahad
f0ffbad77d Fix syncing issue with Notification statuses:
The timeout_notifications job runs daily and does not correctly
update both the status fields to keep them in sync.
2017-07-06 12:30:08 +01:00
Rebecca Law
5ef0450e37 Update the json in the post to the service for the inbound sms. 2017-06-22 10:15:08 +01:00
Rebecca Law
c608f5997b New task to send the inbound sms to the service.
If the service has not set the url then nothing happens.
If the request to the service url returns with 500 or greater the task is retries.
The task is created when the SMS provider post the inbound SMS.
2017-06-20 17:13:40 +01:00
Imdad Ahad
1dacc6c3a3 Updates:
* Skip the decorated tests as that doesn't work as we expect
* Remove magic numbers in tests
2017-06-13 15:22:31 +01:00