Commit Graph

27 Commits

Author SHA1 Message Date
Leo Hemsted
92d78956be Merge pull request #2592 from alphagov/reporting-worker
Add reporting worker
2019-08-15 17:22:27 +01:00
Leo Hemsted
e5c76ffda7 reduce days to process from 10 to 4
to try and speed it up temporarily.
2019-08-15 17:06:38 +01:00
Leo Hemsted
58f24a0a83 trigger nightly delete tasks from the create notification status task
the nightly tasks need to run after the create nightly notification
status task - so that test notifications are still there to record
stats for, and to stop the risk of deleting notificaitons part-way
through recording stats for them.
2019-08-14 18:04:45 +01:00
Rebecca Law
996dcdd88c Increase the number of days we rebuild the tables for 2019-07-18 16:45:27 +01:00
Rebecca Law
e3ee99e70d Reduce the number of days to recalculate billing. It's not necessary to calculate longer than 4 days. 2019-05-15 14:40:53 +01:00
Rebecca Law
1c68e0f565 Remove unused method.
last_n_days was only being used in a test.
2019-04-12 10:26:46 +01:00
Leo Hemsted
1dc084be54 fix nightly ft stats tables task to respect BST
the create_nightly_notification_status task runs at 00:30am UK time,
however this means that in summer datetime.today() will return the
wrong date as the server (which runs on UTC) will run the task at
23:30 (populating the wrong row in the table).

fix this to use nice tz aware functions
2019-04-02 15:15:07 +01:00
Leo Hemsted
754c65a6a2 create cronitor decorator that alerts if tasks fail
make a decorator that pings cronitor before and after each task run.
Designed for use with nightly tasks, so we have visibility if they
fail. We have a bunch of cronitor monitors set up - 5 character keys
that go into a URL that we then make a GET to with a self-explanatory
url path (run/fail/complete).

the cronitor URLs are defined in the credentials repo as a dictionary
of celery task names to URL slugs. If the name passed in to the
decorator  isn't in that dict, it won't run.

to use it, all you need to do is call `@cronitor(my_task_name)`
instead of `@notify_celery.task`, and make sure that the task name and
the matching slug are included in the credentials repo (or locally,
json dumped and stored in the CRONITOR_KEYS environment variable)
2019-01-18 15:36:53 +00:00
Pea Tyczynska
22ad14fcee Fix logging for create_nightly_notification_status 2018-11-09 11:49:49 +00:00
Pea Tyczynska
987445f1bf ft_notification_status now updates data for 4 days back
This was done so when notification is timed out from sending/pending
to temporary_failure, this change has to always be caught
in the ft_notification_status
2018-11-08 11:52:40 +00:00
Rebecca Law
4fc004b00a Increase the number of days we calculate billing from 3 to 10 days.
Log exception if the billing counts for letters are different in the dvla response file than what we collected.
2018-07-24 16:28:30 +01:00
Rebecca Law
709a6c38c7 Created a task to update ft_notification_status for the last three days. 2018-06-20 16:45:20 +01:00
Rebecca Law
40d8f78b2b Convert the day_start from a string to a datetime. 2018-05-15 14:00:06 +01:00
Rebecca Law
d98581cfe6 Added a new endpoint for yearly usage totals using ft_billing. 2018-05-11 16:25:16 +01:00
Rebecca Law
99d1357c37 Fix the logging message in the nightly task 2018-05-10 17:13:38 +01:00
Rebecca Law
8028f6cc28 We found that the reporting task failed twice because of integrity constraints.
This was because the rate_multiplier was being added as 1 and 1.0 which was not resolving to the same.
This updates the table to use Integrer.
Also changed the logging for the task.
2018-05-10 15:35:58 +01:00
Rebecca Law
d00614205e Adding test for fetch_biling_data_for_day. 2018-04-25 14:45:51 +01:00
Rebecca Law
16ef133aa5 Refactor reporting task so that methods can be used for the annual usage page.
Still a work in progress, tests are coming.
2018-04-25 14:45:51 +01:00
venusbb
52aeec2247 bug fix: use date only when comparing unique records rather than datetime 2018-03-29 11:42:31 +01:00
venusbb
83aa1b9fa3 use datetime.combine rather than parser.parse 2018-03-28 10:52:37 +01:00
venusbb
2262db4f62 Database tweak and BST bug fix 2018-03-27 10:37:56 +01:00
venusbb
55a27fed13 fixed a bug on reporting_task 2018-03-23 11:40:27 +00:00
venusbb
af975b89ec primary key for ft_billing should use notification_type 2018-03-21 15:02:08 +00:00
venusbb
cf019864e2 Preserve 'unknown' in ft_billing for sms if the provider is not known. 2018-03-21 09:50:34 +00:00
venusbb
07aa44ea82 Add import report_tasks to schedule_tasks. 2018-03-20 16:42:24 +00:00
venusbb
bb95a2784f Create schedueled job, fixed tests 2018-03-16 09:22:34 +00:00
venusbb
ea70c6454a Fine-tuning DB model and create taks for data migration
- Removed unused columns in ft_billing
- Create tasks for nightly data migration
2018-03-14 17:07:33 +00:00