Commit Graph

269 Commits

Author SHA1 Message Date
Martyn Inglis
786adb5d71 Move Queuenames in with the celery code, revamp config to allow move to celery 4.x 2017-07-12 12:01:52 +01:00
Martyn Inglis
e0106eb1be hacked celery4.0.2 in. Runs and works
- note though this version of master I branched had split head on sqlalchemey. This needs a new master merge to fix
2017-07-12 11:58:04 +01:00
Leo Hemsted
7f883f1355 don't store non-strings to os.environ
in tests, we were replacing os.environ with a basic dict so that
we didn't overwrite the contents of the real environment during tests.
However, os.environ doesn't accept non-str values, so this commit
changes the fixture so that it asserts all values set are strings.
We needed to change how we store ip whitelist stuff in the env because
of this.
2017-07-11 15:41:44 +01:00
venusbb
5d57189187 changed name of test fixture 2017-07-11 09:50:09 +01:00
venusbb
5089a4d53b retrieve sms ip whitelist from credentials on paas 2017-07-10 17:03:43 +01:00
venusbb
39a8e1cfed IP address in config file uses comma rather than full stops 2017-07-06 12:30:54 +01:00
venusbb
d6c69bf437 Check inbound sms IP address stage 1
IP address checked
2017-07-06 12:29:37 +01:00
Leo Hemsted
ac7665bfc6 celery test cleanup
* Alter config so an error will be raised if you forget to mock out a
  celery call in one of your tests
* Remove an unneeded exception type that was masking errors
2017-06-20 12:06:49 +01:00
Leo Hemsted
88a479a4bb make client request fixture args conform 2017-06-20 12:06:49 +01:00
Imdad Ahad
f29ebb34f6 Add scheduled task to delete response files older than 7 days 2017-06-13 12:40:58 +01:00
Imdad Ahad
34f4829362 Merge pull request #1025 from alphagov/imdad-feat-remove-dvla-transformed-files
Remove transformed DVLA files older than 7 days
2017-06-13 10:23:31 +01:00
minglis
1b0ff63b1e Merge pull request #1022 from alphagov/remove-nasty-query-from-dashboard
Remove nasty query from dashboard
2017-06-08 10:58:35 +01:00
Imdad Ahad
c4a89b77e3 Add task to remove transformed dvla files:
* Runs at 4:40am
* Remove files older than 7 days and only two day's worth
2017-06-07 16:39:23 +01:00
Imdad Ahad
74a8905be9 Seperate deletion of jobs:
* Two separate jobs, one for sms&email and another for letter
* Change celery task for delete to accept template type filter
* General refacor of tests to make more readable
2017-06-06 16:02:01 +01:00
Martyn Inglis
75bf693f44 Add the yearly free limit to the service model.
This allows us to reference it across the API code base and return it in the API.

But not currently attached to the service DB model - a static method on the class.
2017-06-06 14:49:05 +01:00
Imdad Ahad
56c3f3cf7c Add task to delete inbound sms everyday at 1am 2017-06-02 16:10:27 +01:00
Chris Hill-Scott
eb493f6817 Rename ‘One-off message’ to ‘Report’
See 7915845cb5
2017-06-01 13:56:47 +01:00
Chris Hill-Scott
78d071f22f Ignore one-off messages in job list on dashboard
Same as how we ignore ‘send yourself a test’ messages (see:
d8467bfc3c). The dashboard gets clogged
up with one off messages otherwise, which affects:
- performance
- users ability to find their jobs
2017-06-01 12:30:47 +01:00
Martyn Inglis
d74675b6d8 Removed references to old queus 2017-05-30 11:24:26 +01:00
Martyn Inglis
75c5e5e5d5 Merge branch 'master' into redo-queue-visibitlity-timeout
Conflicts:
	app/notifications/process_notifications.py
	app/v2/notifications/post_notifications.py
2017-05-30 10:18:18 +01:00
Martyn Inglis
29b6a918e8 Ensure dev and test builds will read from old and new queues 2017-05-25 13:43:31 +01:00
Martyn Inglis
58af4e5ee9 timeout slightly longer than retry 2017-05-25 11:20:23 +01:00
Martyn Inglis
21586c917c Created an object to hold queue names.
- This is to be used throughout the app.
- Allows us to manage queue names centrally
- All queues renamed to allow us to change the retry processing/visibility timeout
2017-05-25 10:50:55 +01:00
Rebecca Law
9f6c037530 Use iso8601 to validate scheduled_for datetime.
Added a validation method that always fails for scheduled notifications.
Comment out config for scheduled task.
The schedule notifications will be turned on once we can invite services to use it.
Waiting for the service permission story, must commit this in order to keep things from going stale.
2017-05-24 16:27:15 +01:00
Martyn Inglis
07b527bb1b All queues now managed in the same way:
- TEST and DEV builds set up their own queue lists for reading
- PROD/STAGE/PREVIEW use the separate worker process with the -Q flag

This enables us to rename queues in due course
2017-05-24 15:47:20 +01:00
Rebecca Law
96b0e42959 Merge branch 'master' into schedule-api-notification
Conflicts:
	app/celery/scheduled_tasks.py
	tests/app/celery/test_scheduled_tasks.py
2017-05-24 13:21:22 +01:00
minglis
8a2715e3e1 Merge pull request #980 from alphagov/change-delete-jobs-to-delete-by-type
Change delete jobs to delete by type
2017-05-24 12:35:04 +01:00
Rebecca Law
53efb4e033 Merge branch 'master' into schedule-api-notification
Conflicts:
	tests/app/v2/notifications/test_post_notifications.py
2017-05-23 13:58:50 +01:00
Martyn Inglis
a751637eaa Added the schedule into celery beat.
run the delete jobs 20 mins apart,
00:00 -> sms
00:20 -> email
00:40 -> letters
2017-05-23 13:41:22 +01:00
Leo Hemsted
3744463296 treat 40604 and GOVUK as not having a sender ID
in prep for removing the 40604-as-default, first we need to make sure
that if you either have GOVUK or None as your sms sender, then we send
GOVUK through to the provider
2017-05-22 17:10:07 +01:00
Rebecca Law
76d0783c63 Add beat config for send-scheduled-notifications task to run every 15 minutes.
Added the missing commit to the update pending
2017-05-22 16:30:45 +01:00
Martyn Inglis
8dc7a86148 Merge branch 'master' into async-job-stats
Conflicts:
	app/dao/services_dao.py
2017-05-18 09:34:54 +01:00
minglis
05a179c6ef Merge pull request #941 from alphagov/load-service-on-auth
Load service on auth
2017-05-17 10:49:00 +01:00
Martyn Inglis
83a1b1526e Merge branch 'master' into async-job-stats
Conflicts:
	app/notifications/notifications_ses_callback.py
2017-05-17 09:49:43 +01:00
Imdad Ahad
570d0ec9db Send notification to active service users with user fields (optional) 2017-05-12 14:06:29 +01:00
Martyn Inglis
02f7abcc48 Wire in the new task. Runs at 5am after the other tasks. 2017-05-11 15:23:45 +01:00
Martyn Inglis
a4549a2e26 Put the statistics tasks into a new queue. This currently has no worker in PaaS so blocks the deployment until new PaaS worker is configured. 2017-05-09 22:03:13 +01:00
Martyn Inglis
07daa369a3 Fixed some config issues 2017-05-05 15:20:34 +01:00
Martyn Inglis
97dcf766bc Switch on API rate limiting on production.
Limits POST API calls to send notifications on V1/V2 to 3000 in 60 seconds.
2017-05-03 12:13:32 +01:00
Martyn Inglis
a9539d892c Merge branch 'master' into rate-limit-api-calls
Conflicts:
	requirements.txt
	tests/app/notifications/rest/test_send_notification.py
	tests/app/notifications/test_validators.py
	tests/app/v2/notifications/test_post_notifications.py
2017-05-02 10:56:56 +01:00
Leo Hemsted
02e8f454e1 add new notification status 'sent'
for international text messages, when we may not ever receive callback info,
so we need a separate status that doesn't get timed out etc
2017-04-25 17:08:29 +01:00
Martyn Inglis
803c21865b Allow rate limiting on a per env basis
- switched off on prod by default
2017-04-25 09:54:09 +01:00
Martyn Inglis
926b8a60f9 Adds in call to new rate limit method in the redis client
- both V1 and V2 APIs
- Rate limiting wrapped into a new method - check_rate_limiting
	- delegates to the previous daily limit and the new though put limit
- Rate limiting done on key type. Each key has it's own limit (number of requests) and interval (time period of requests)
- Configured in the config. Not done on a per-env basis though could be in the future.
2017-04-24 14:15:08 +01:00
Martyn Inglis
b08ac6eb63 Ensure priority queue read by development machines. It's now a default queue in the config. 2017-04-04 13:53:01 +01:00
Martyn Inglis
787d0da7db Broke the periodic queue out from the standard worker.
- now has own worker with worker count of 2
- this is too ensure that any issues with scheduled jobs do not affect core applications
2017-04-04 11:23:54 +01:00
Martyn Inglis
8f5e1318eb Spread out the scheduled jobs.
Before the longer tasks all ran on the hour, at midnight, 1 and 2.

Now we run one at midnight, one at quarter past, half past then at quarter too. Trying to spread the load.
2017-04-03 11:17:27 +01:00
Ken Tsang
316d8fdb0d Updated config to cloudfoundry_config 2017-03-31 16:37:04 +01:00
Ken Tsang
c633e193b0 Update config to enable performance platform 2017-03-31 15:54:29 +01:00
Ken Tsang
989713b0f1 Update config for daily perf job to start 4am 2017-03-31 15:33:41 +01:00
bandesz
974b59a5b7 Add 'production' alias for live environment 2017-03-21 14:43:56 +00:00