Commit Graph

168 Commits

Author SHA1 Message Date
Leo Hemsted
3331491ef1 add pycurl to requirements
it's needed by kombu 4.0.2.

Also moved import about
2017-07-12 17:17:02 +01:00
Leo Hemsted
1a03248317 temp fix to sort out circular imports 2017-07-12 13:02:19 +01:00
Martyn Inglis
0e9e8955f7 Finished celery refactor - set up config for queue prefix
LEO notes: Also made sure the Test BROKER_URL is preserved so that
tests warn you when celery isn't mocked out
2017-07-12 12:37:18 +01:00
venusbb
d6c69bf437 Check inbound sms IP address stage 1
IP address checked
2017-07-06 12:29:37 +01:00
venusbb
16571c5b3c IP restriction for inbound sms test 2 2017-06-29 10:47:27 +01:00
Leo Hemsted
ef52337d85 add inbound sms api
two endpoints:
* get all inbound sms for a service (you can limit to the X most
  recent, or filter by user's phone number [which will be normalised])
* get a summary of inbound sms for a service - returns the count of
  inbound sms in the database, and the date that the most recent was
  sent
2017-06-02 15:20:18 +01:00
Martyn Inglis
34c9198d0c Make the service available to code on the request context 2017-05-05 15:20:23 +01:00
Martyn Inglis
e977e7cee4 Wired in a simple callback to handle SNS notifications from S3
S3 will send a message when a file lands - which will trigger processing of DVLA responses.
2017-04-25 14:56:16 +01:00
Imdad Ahad
c4fac1d937 Revert "Revert "add DVLA organisations to API"" 2017-04-21 16:05:07 +01:00
Martyn Inglis
5550201939 Upper case only in the random notification id generator
as DVLA need upper case only or the barcode doesn't generate.
2017-04-20 10:55:53 +01:00
Ken Tsang
89e244ccd2 First commit 2017-04-19 10:51:12 +01:00
Martyn Inglis
b0e5062df2 Added the random string reference to the letter
- uses the reference field on the notifications table to store a 16char random string used to cross reference DVLA letters back to the notification
- used as letter barcode does not have space for a UUID notification id

Depends on https://github.com/alphagov/notifications-utils/pull/149

Renamed the numeric_id to notification_reference in utils and changed validation rules to match this

Note also the persist_notification method set "reference" to be "client_reference" which is confusing and they are different things, so fixed this too.
2017-04-12 17:56:55 +01:00
Rebecca Law
970a4e7b4e New endpoint to send a list of job ids to a queue.
The task will be picked up by the FTP app. Given the list of job ids the tasks will get all the files from s3, aggregate them then send to dvla
2017-04-06 12:14:17 +01:00
Ken Tsang
8e5266e89b Refactor and fix auth errors 2017-03-22 10:54:15 +00:00
Ken Tsang
b81d789307 Add preview endpoint and tests 2017-03-22 10:11:31 +00:00
Rebecca Law
37293b6c7a Added a test to ensure that all blueprints have a registered before_request method.
Added a test for the status endpoint.
2017-03-17 16:21:41 +00:00
Rebecca Law
78242812ef Register a before_request event for all blueprints, that defines the authentication requirement.
There are three authentication methods:
 - requires_no_auth - public endpoint that does not require an Authorisation header
 - requires_auth - public endpoints that need an API key in the Authorisation header
 - requires_admin_auth - private endpoint that requires an Authorisation header which contains the API key for the defined as the client admin user
2017-03-16 18:15:49 +00:00
Rebecca Law
77f520acba Created an endpoint to test how the incoming messages from MMG will work.
So this just prints the response to logs, removing the phone number first. Then returns the requested RECEIVED.
2017-03-06 11:58:49 +00:00
Imdad Ahad
c811f1b6c6 Init the perf platform client, add logs and refactor payload methods 2017-01-30 18:24:06 +00:00
Imdad Ahad
ed52f41039 Add, init and config performance platform client to prepare/upload daily notification stats 2017-01-27 12:21:08 +00:00
bandesz
fd66fbd719 Run API on Paas 2017-01-13 10:24:13 +00:00
Martyn Inglis
7260561ad5 Makes the app use the redis and statsd clients from utils 2016-12-01 17:20:05 +00:00
Martyn Inglis
7fe56cf992 Add request ID logging to the API.
- adds a request ID if needed to use in all log events
- respects request-ids passed in from calling clients
2016-11-30 17:05:39 +00:00
minglis
40cf6c049f Merge pull request #746 from alphagov/record-user-agent
Statsd counter for user agents.
2016-11-30 12:51:30 +00:00
Martyn Inglis
e698d124ba Statsd counter for user agents.
We want to log the usage of the various API clients we have so that we understand when they can be cycled.

To this end we are going to count usage in statsd.

All notify clients have a suer agent, of the format: NOTIFY-API-{LANGUAGE}-CLIENT/version.number

For example, NOTIFY-API-PYTHON-CLIENT/3.0.0

We convert that into a statsd/graphite friendly string of the format: notify-api-python-client.3-0-0

So we can subdivide on client and client version on our dashboards.

Present but unknown User agents are records as "non-notify-user-agent"

Missing are presented as "unknown"
2016-11-25 13:27:36 +00:00
Rebecca Law
669126d41b Merge pull request #744 from alphagov/add-default-exception-handler
Add application level error handler.
2016-11-25 09:34:23 +00:00
Rebecca Law
52c8596c92 Add application level error handler.
This should catch any unexcpeted exceptions.
2016-11-24 17:09:56 +00:00
Martyn Inglis
2a2ef34339 Merge branch 'master' into caching-with-redis
Conflicts:
	app/__init__.py
2016-11-23 09:12:11 +00:00
Paul Craig
c1fa5e156a Append "Z" to DATETIME_FORMAT
We're formally using the ISO 8601 UTC datetime format, and so the
correct way to output the data is by appending the timezone.
("Z" in the case of UTC*).

Unfortunately, Python's `datetime` formatting will just ignore the
timezone part of the string on output, which means we just have to
append the string "Z" to the end of all datetime strings we output.

Should be fine, as we will only ever output UTC timestamps anyway.

* https://en.wikipedia.org/wiki/ISO_8601#UTC
2016-11-21 15:59:10 +00:00
Paul Craig
9758b96a2b Create 'v2' get notification route
The new 'v2' API wants to return less data than the previous one,
which was sending back tons of fields the clients never used.

This new route returns only useful information, with the JSON
response dict being built up in the model's `.serialize()` method.

Note that writing the test for this was a bit painful because of
having to treat loads of keys differently. Hopefully we think this
is a good way to write this test, because if we don't, we should
start thinking of a better way to check the values are what we
expect.
2016-11-21 15:41:49 +00:00
Martyn Inglis
6c27f8023e Added tests for setting and fetching from redis respecting the enabled flag. 2016-11-10 13:22:38 +00:00
Martyn Inglis
aabda3f83e Refactored the initialisation of the redis client to follow the model used in the StatsdClient.
Basically wrapped the client so we can enable/disable, exception handle and so on.
2016-11-10 11:50:49 +00:00
Martyn Inglis
eb94fe6c0a Adding Flask-Redis to the project.
[https://pypi.python.org/pypi/Flask-Redis/0.1.0](https://pypi.python.org/pypi/Flask-Redis/0.1.0)

Initial addition as we think about redis as cache.
2016-11-10 11:27:57 +00:00
Martyn Inglis
aea2356ae1 Initial SPIKE into redis for cache 2016-11-07 11:55:59 +00:00
Rebecca Law
c2eecdae36 - Add validation methods for post notification.
- Use these validation methods in post_sms_notification and the version 1 of post_notification.
- Create a v2 error handlers.
- InvalidRequest has a to_dict method for private and v1 error responses and a to_dict_v2 method to create the v2 of the error responses.
- Each validation method has extensive unit tests, so the unit test for the endpoint do not need to check every error case, but check that the error handle formats the message correctly.
- The format of the error messages is still a work on progress.
- This version of the api could be deployed without causing a problem to the application.
- The new endpoing is still a work in progress and is not being used yet.
2016-10-27 11:46:37 +01:00
Martyn Inglis
37a2dc7925 Implemented the REST endpoint to communicate with the new send-to-provider code
- this allows us to send a notification to a provider by means of an API call
- This is in addition to the celery code.
- idea is that we can use this method to help speed up throughput by generating API traffic by node/lambda etc to supplement the celery code in times of high load.
2016-09-22 14:01:25 +01:00
Martyn Inglis
c1b00a5f0c Simplified the config. Aim is to get the actual secrets in credstash to be env specific, and not the random collection of things we have at the moment.
Secret definition also includes env specific things such as URLs / Queue prefixes / URLs for providers and so on.
2016-09-07 09:35:31 +01:00
Leo Hemsted
2053ebd933 rename service to service_blueprint
it was causing a bug where a local variable service was not being
instantiated and we were trying to operate on the blueprint instead

it's being used in so few places it makes sense to rename it
2016-08-24 15:00:51 +01:00
Leo Hemsted
f065b08db2 remove unused dao functions 2016-08-11 16:13:53 +01:00
Leo Hemsted
fff81b4910 remove unused notification-statistics endpoints 2016-08-11 11:55:28 +01:00
Leo Hemsted
e631333a34 add GET /organisation and GET /organisation/id endpoints 2016-08-09 15:59:22 +01:00
Martyn Inglis
0fa614d812 Removed unused import 2016-08-08 11:09:07 +01:00
Martyn Inglis
f223446f73 Refactor statsd logging
Removed all existing statsd logging and replaced with:

- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables

- statsd client changed to prefix all stats with "notification.api."

- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way

	notifications.api.POST.notifications.send_notification.200

This should allow us to aggregate to the level of

	- API or ADMIN
	- POST or GET etc
	- modules
	- methods
	- status codes

Finally we count the callbacks received from 3rd parties to mapped status.
2016-08-05 10:44:43 +01:00
Martyn Inglis
dbe34f46e6 Reordered imports 2016-07-29 10:15:34 +01:00
Martyn Inglis
9ec52eac02 New config style 2016-07-29 10:15:34 +01:00
Rebecca Law
f52755742c Split send_email task into one task to create the notification and one to send it to the provider.
Is there is an exception the task will go to the retry queue.
2016-07-01 14:14:28 +01:00
Leo Hemsted
11093530f9 Merge pull request #485 from alphagov/api_user-cleanup
Api user cleanup
2016-07-01 10:58:18 +01:00
Leo Hemsted
18b30de452 fix calling init_app twice
this was causing flask decorators like auth check to be mounted twice
2016-06-30 10:44:21 +01:00
Chris Hill-Scott
7c3d25a87a Publish a Swagger specification
Adds a new endpoint (`/spec`) which returns a the specification of the
API in Swagger-formatted JSON.

This means we will have something to point frontends at, so we can
evaluate which ones we like.

Right now it’s all hand-defined. If we were consistent about our use of
Marshmallow we could generated the spec from the Marshmallow schemas.
2016-06-29 12:07:32 +01:00
Rebecca Law
f8262b13c1 Refactor permissions dao - removed custom query filter
Removed permissions rest - not being used.
2016-06-15 16:32:52 +01:00