Commit Graph

230 Commits

Author SHA1 Message Date
Martyn Inglis
19c14982a2 Call the task wrapper outcome function in the statistics_tasks file. This wraps the logic around which tasks to creates and simplifies the logic in the Callback classes. 2017-05-09 22:03:57 +01:00
Martyn Inglis
caed193647 Use the new task wrapper methods rather than creating a task directly 2017-05-09 18:17:55 +01:00
Sakis
01e3a88d55 Merge pull request #920 from alphagov/autoconfirm_http_subscriptions
Autoconfirm http subscriptions
2017-05-02 17:03:41 +01:00
Martyn Inglis
b0baa1532a Added some logging around rate limiting. 2017-05-02 11:14:45 +01:00
Martyn Inglis
a9539d892c Merge branch 'master' into rate-limit-api-calls
Conflicts:
	requirements.txt
	tests/app/notifications/rest/test_send_notification.py
	tests/app/notifications/test_validators.py
	tests/app/v2/notifications/test_post_notifications.py
2017-05-02 10:56:56 +01:00
Athanasios Voutsadakis
cfed90e502 Make code a bit more defensive and add som logging 2017-04-28 16:10:41 +01:00
Athanasios Voutsadakis
74433c9335 Add tests for autoconfirmation 2017-04-28 15:11:36 +01:00
Athanasios Voutsadakis
04b003c152 Add utility function for subscription autoconfirm 2017-04-28 15:11:36 +01:00
Imdad Ahad
252c3235d7 Persist intl fields on notification 2017-04-27 13:24:36 +01:00
Martyn Inglis
dfbca8043e Fixed typo errors 2017-04-26 21:08:43 +01:00
Martyn Inglis
2a0f8c8808 Validate International phone numbers
- uses new utils methods to validate phone numbers
- defaults to International=True on validation. This ensures the validator works on all numbers
- Then check if the user can send this message to the number internationally if needed.
2017-04-26 15:56:45 +01:00
Martyn Inglis
ad8e445558 Reformatted code 2017-04-26 10:24:39 +01:00
Martyn Inglis
e977e7cee4 Wired in a simple callback to handle SNS notifications from S3
S3 will send a message when a file lands - which will trigger processing of DVLA responses.
2017-04-25 14:56:16 +01:00
Martyn Inglis
803c21865b Allow rate limiting on a per env basis
- switched off on prod by default
2017-04-25 09:54:09 +01:00
Martyn Inglis
926b8a60f9 Adds in call to new rate limit method in the redis client
- both V1 and V2 APIs
- Rate limiting wrapped into a new method - check_rate_limiting
	- delegates to the previous daily limit and the new though put limit
- Rate limiting done on key type. Each key has it's own limit (number of requests) and interval (time period of requests)
- Configured in the config. Not done on a per-env basis though could be in the future.
2017-04-24 14:15:08 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Martyn Inglis
ac445114c7 When using a priority template put the notification in the priority queue.
- there is now a dedicated worker to handle priority queues. We don't now bundle this in the default worker.
2017-04-04 13:44:11 +01:00
Rebecca Law
3838fb583c The redis cache for daily limits and template usage was being updated for notifications with a test api key.
This PR corrects that.
2017-03-30 13:43:44 +01:00
Martyn Inglis
7dcd3164e2 Revert "Reinstating the 2 task model for API submitted notifications."
This reverts commit 1c154c4113.
2017-03-30 10:46:23 +01:00
Martyn Inglis
385a73da2b Revert "ensure we're passing through api keys and key types from notifications"
This reverts commit 25d1777937.
2017-03-30 10:38:41 +01:00
Leo Hemsted
25d1777937 ensure we're passing through api keys and key types from notifications
when we made the change to async persist notifications, we forgot to
pass through api_key_id and key_type. in send_sms/email, for legacy
reasons, they default to None/KEY_TYPE_NORMAL, so regardless of what
your api key was set up as, we would send real messages!

TODO: Once the PaaS transition is complete and the task changes are
reverted, remove the api_key_id and key_type params from the send_*
tasks entirely, as those are only called from the csv job flow, and
don't need them
2017-03-28 13:14:46 +01:00
Martyn Inglis
1c154c4113 Reinstating the 2 task model for API submitted notifications.
This is being done for the PaaS migration to allow us to keep traffic coming in whilst we migrate the database.

uses the same tasks as the CSV uploaded notifications. Simple changes to not persist the notification, and call into a different task.
2017-03-23 14:41:00 +00:00
Rebecca Law
37293b6c7a Added a test to ensure that all blueprints have a registered before_request method.
Added a test for the status endpoint.
2017-03-17 16:21:41 +00:00
Rebecca Law
78242812ef Register a before_request event for all blueprints, that defines the authentication requirement.
There are three authentication methods:
 - requires_no_auth - public endpoint that does not require an Authorisation header
 - requires_auth - public endpoints that need an API key in the Authorisation header
 - requires_admin_auth - private endpoint that requires an Authorisation header which contains the API key for the defined as the client admin user
2017-03-16 18:15:49 +00:00
Chris Hill-Scott
761ff86591 Merge pull request #853 from alphagov/ignore-additional-personalisation
Quietly ignore extra personalisation
2017-03-10 16:16:27 +00:00
Rebecca Law
51ab7a5dbf Change log message 2017-03-08 14:40:12 +00:00
Chris Hill-Scott
e507fed152 Quietly ignore extra personalisation
> If a user makes an API request with additional personalisation fields,
> we should simply discard any fields that the template doesn't have.
>
> This gives a couple of related advantages:
>
> - modifying template parameters no longer requires downtime for
>   clients - as they can pass in extra new parameters before a template
>   change, or continue passing in old unused parameters after removing
>   them from a template
>
> - services can pass in large user objects, for example, and then play
>   around with templates adding and removing fields at will
>
> we should make sure we still return an error if a user doesn't pass in
> a required parameter.

– https://www.pivotaltracker.com/story/show/140774195
2017-03-07 16:09:17 +00:00
Rebecca Law
77f520acba Created an endpoint to test how the incoming messages from MMG will work.
So this just prints the response to logs, removing the phone number first. Then returns the requested RECEIVED.
2017-03-06 11:58:49 +00:00
bandesz
271b664a7e Log SNS subscription URL 2017-03-03 12:17:53 +00:00
Rebecca Law
74e29708f9 Fix bug where the increment calls set count to 1 if the cache does not exist. 2017-02-15 11:49:19 +00:00
Rebecca Law
53b7ad0961 Moved the cache key to the utils module.
Renamed the dao method.
2017-02-14 14:22:52 +00:00
Rebecca Law
458adefcb8 Added a redis cache for the template usage stats.
Cache expires every 10 minutes, but will help with the every 2 second query, especially when a job is running.
There is some clean up and qa to do for this yet
2017-02-13 18:47:29 +00:00
Rebecca Law
c6eb284218 Small change to include the reference that ses send us on the callback. 2017-01-30 14:09:30 +00:00
Rebecca Law
27b6aa736c Update info log to waring.
Removed duplicate test
2017-01-27 12:20:54 +00:00
Rebecca Law
67657ced26 Return success of the callback is a duplicate or for an id that does not exist
This PR changes the response to POST /notifications/sms/<mmg | firetext> from a 400 response to a 200 response.
If we get a callback for a notification more than once or for a notification we log that but we return a 200 success response to the provider.
We have found that there is a situation where the send to provider throws a timeout exception but the provider did get the message, but we still send it to them again.
In which case they send the message twice, and callback for the message twice.
Another case where we may get duplicate callbacks is that the network gave the provider two callbacks meaning they pass those two callbacks onto us.

So it is really difficult to know if we sent to the provider twice or just got two callbacks.

The test_callback has many changes because I took the opportunity to use the client conftest fixture rather than the notify_api fixture.
The only 2 tests really changed are test_mmg_callback_returns_200_when_notification_id_not_found_or_already_updated and test_firetext_callback_returns_200_when_notification_id_not_found_or_already_updated
2017-01-25 14:37:11 +00:00
Rebecca Law
8ed0979251 increase page size for API calls to 250 2017-01-20 12:26:55 +00:00
Rebecca Law
f66670c558 - Added logging to indicate that we created the notification and that we sent the notification to a delivery queue.
- Small updates as recommended by review comments.
2017-01-18 09:56:26 +00:00
Rebecca Law
b34c5f0bcd If the template.process_type = PRIROITY send the message to the notify queue.
We are using the notify queue in this iteration because that queue is a low volume queue with it's own dedicated workers. This just saves us from building a new queue at this point, and a new queue may not be necessary.
2017-01-17 13:16:26 +00:00
Rebecca Law
a8cdabcecd - Refactor v2 post_notification to use a single method for sms and email.
- Added the `simulate` notification logic to version 2. We have 3 email addresses and phone numbers that are used
to simulate a successful post to /notifications. This was missed out of the version 2 endpoint.
- Added a test to template_dao to check for the default value of normal for new templates
- in v2 get_notifications, casted the path param to a uuid, if not uuid abort(404)
2017-01-17 12:08:24 +00:00
Martyn Inglis
0f37824b0c Ensure updates on a research mode service or test key don't touch the history table
- note this is an unexpectedly big change.
- When we create a service we pass the service id to the persist method. This means that we don't have the service available to check if in research mode.
- All calling methods (expecting the one where we use the notify service) have the service available. So rather than reload it I changed the method signature to pass the service, not the ID to persist.
- Touches a few places.

Note this means that the update or create methods will fall over on a null service. But this seems correct.

Goes back to the story which we need to play to make the service available as the API user so that the need to load and pass around services is minimised.
2016-12-19 16:45:18 +00:00
Rebecca Law
be9861c867 Merge pull request #769 from alphagov/add-log-to-ses-callback
Added logging when the SES callback is successful.
2016-12-16 11:58:43 +00:00
Rebecca Law
7a623de171 Added boby and subject to the get_notifications schema 2016-12-15 16:19:55 +00:00
Rebecca Law
74a4651de2 Added logging when the SES callback is successful. 2016-12-15 10:34:29 +00:00
Chris Hill-Scott
59af44d7ab Update utils to 12.0.0
Includes:

- [x] https://github.com/alphagov/notifications-utils/pull/94 (breaking
      changes which are responsible for all the changes to the API in
      this PR)

The test for `get_sms_fragment_count` has been removed because this
method is already tested in utils here:

ac20f7e99e/tests/test_base_template.py (L140-L159)
2016-12-13 10:57:01 +00:00
Jenny Duckett
d5d079a150 Add optional queue param to send_notification_to_queue
We want to use this function for sending internal notifications to a different
queue.
2016-12-09 16:46:49 +00:00
Chris Hill-Scott
d449475dd5 Incorporate breaking utils changes
The `.replaced…` methods on instances of `Template` were removed in
https://github.com/alphagov/notifications-utils/pull/84
2016-12-02 10:42:11 +00:00
Martyn Inglis
7260561ad5 Makes the app use the redis and statsd clients from utils 2016-12-01 17:20:05 +00:00
minglis
0ddd8d93ae Merge pull request #753 from alphagov/logging-service-ids
Records the service ID if present in the request
2016-12-01 11:38:49 +00:00
Martyn Inglis
23ca62a7ac Records the service ID if present, or if an ADMIN api call the string admin-api.
this is used later in logging of requests.
2016-11-30 10:59:55 +00:00
Rebecca Law
3267a54ded Merge branch 'master' into dont-send-message-twice 2016-11-28 16:43:17 +00:00