Commit Graph

815 Commits

Author SHA1 Message Date
Martyn Inglis
805da23d6b Starting to implement scheduling 2016-08-24 14:04:52 +01:00
Martyn Inglis
5adecda41e Adds new job_status table and FK to jobs.
This will replace the job_status enum type.
2016-08-24 13:34:42 +01:00
Martyn Inglis
acf3c568db Tidied up the query indentation 2016-08-24 10:50:29 +01:00
Martyn Inglis
09cb94081b Returns the outcome statistics for the job on the API call. 2016-08-23 16:46:58 +01:00
Leo Hemsted
26d7675baa pep8 fixes
no idea why the build/local pep8s weren't picking them up before.

also excluded import order pep8
2016-08-23 12:05:47 +01:00
Leo Hemsted
132addb181 Merge branch 'master' into noti-stats-cleanup 2016-08-23 09:49:12 +01:00
Martyn Inglis
fe22c876b6 Merge branch 'master' into template_useage_per_id 2016-08-23 09:32:01 +01:00
Andras Ferencz-Szabo
0b73da94e7 Merge pull request #605 from alphagov/jenkins_deploy
Create Docker build image, build project with Docker
2016-08-22 17:57:39 +01:00
bandesz
68c1135379 Create Docker build image, build project with Docker 2016-08-22 17:25:41 +01:00
Martyn Inglis
4e6da1ba55 Changed template stats for template id
- now returns the most recent notification history row for that template ID.
- contains all the required data for the use cases for that template
2016-08-22 14:35:04 +01:00
Martyn Inglis
fdd85b7dc0 Merge branch 'master' into template_queryies 2016-08-22 11:42:26 +01:00
Leo Hemsted
5418f65ae7 Merge pull request #599 from alphagov/version-500
fix GET /notifications 500 error
2016-08-22 10:59:46 +01:00
Martyn Inglis
b7476a1975 Removed the group by day aspects of template stats. Not needed. Grouped by template only. 2016-08-22 10:38:44 +01:00
Martyn Inglis
9eb559d4b2 Renamed the param to avoid shadowing 2016-08-18 15:29:56 +01:00
Martyn Inglis
ede7d0cbea Removed old endpoint.
Going to handle the migration in the clients.
2016-08-18 14:06:12 +01:00
Martyn Inglis
7a5acea71b New endpoint for template stats to use new query. Breaking change to the returned JSON, so adding on a different url. 2016-08-18 14:01:31 +01:00
Martyn Inglis
f74000f548 Add query to get template usage from the Notifications History table
- groups by template Id and Day.

Returns count per day, template name, template id, template type, and day.

Ordered by day (desc) and template name (acc)
2016-08-18 12:06:00 +01:00
minglis
7fab5ada38 Merge pull request #596 from alphagov/extend-timeout-on-db-retries
Improve logging and extend time periods associated with retrying
2016-08-17 09:24:49 +01:00
Leo Hemsted
721929ca75 Merge pull request #589 from alphagov/org-rendering
Render organisational branding in emails
2016-08-16 13:26:09 +01:00
Leo Hemsted
5491668579 let users set organisation on POST /service/{id} 2016-08-15 10:54:26 +01:00
Leo Hemsted
cc7ea8043c add organisation and branding to GET /service response dict 2016-08-12 11:40:57 +01:00
Leo Hemsted
5b3a0f03d3 rename actual_template to template_history
it's slightly less emotionally charged
2016-08-12 10:29:23 +01:00
Leo Hemsted
8fb0ba56ed unused schema 2016-08-11 17:30:50 +01:00
Leo Hemsted
b22e3845aa kill unused schema 2016-08-11 16:34:36 +01:00
Leo Hemsted
f065b08db2 remove unused dao functions 2016-08-11 16:13:53 +01:00
Leo Hemsted
fff81b4910 remove unused notification-statistics endpoints 2016-08-11 11:55:28 +01:00
Chris Hill-Scott
ebfaa5dac2 Show separate error messages for team key
Although using a team key is functionally the same as your service being
restricted, conflating the two errors is not helpful. What we typically
saw in research was that someone was using a team key, got the error,
used a live key and got the _same_ error.

This commit adds a new error message that specifically mentions the type
of API key that you’re using.
2016-08-10 16:20:01 +01:00
Chris Hill-Scott
0a429e18b4 Give better error when sending to non-team member
Scenario we saw in research:
- trying to send a message to someone outside your team
- service is in trial mode

Result:
- error message was terrible, no-one understood it

Solution:
- better error message
2016-08-10 16:17:47 +01:00
Leo Hemsted
4dd924d438 Merge branch 'master' into version-500 2016-08-10 14:55:07 +01:00
Leo Hemsted
52b2f4f583 require fields in marshmallow schema to try and make it as hard
to break backwards compatability as possible
2016-08-10 11:27:28 +01:00
Martyn Inglis
67a4ee7d51 In event of a task retry ensure the log message is identifier as such
- "RETRY" prefixes the messages

In event of the retry attempts completing without successfully completing the task identify message as such

- "RETRY FAILED" prefixes the messages

Applies to the send_sms|send_email and send_sms_to_provider|send_email_to_provider tasks

These are there to try and ensure we can alert on these events so that we know if we have started retrying messages

Retry messages also contain notification ids to aid debugging.
2016-08-10 08:53:15 +01:00
Martyn Inglis
5b8fbada11 Extend the retry time on the create notification DB tasks
- Was previoulsy 5 attempts 5 seconds apart
- No 5 times 5 minutes apart.
- Gives 25 mins of retries before an error
2016-08-10 08:46:37 +01:00
Leo Hemsted
5b7af00295 re-add content_char_count to public GET /notification endpoints
we shouldn't remove stuff from endpoints or we'll break clients
2016-08-09 17:30:38 +01:00
Leo Hemsted
4ba3745159 update schema to use template_history for accurate template details
only in the public notification endpoint so far for fear of breaking
things - in an ideal world i'd remove the template relationship
from models entirely and replace that with actual_template
2016-08-09 16:57:18 +01:00
Leo Hemsted
46c0728b12 add actual_template relationship to notification
also renamed the function to make it apparent that it'll join and grab personalisation
2016-08-09 16:57:18 +01:00
Leo Hemsted
c820938ced fix schema and primary key
* version is an additional primary key so we need to indicate that
* schema no longer relies on Template model, and uses nested user
2016-08-09 16:56:43 +01:00
Leo Hemsted
049514d4b2 remove history-meta for templates, replace with hand-made history table
history-meta's dynamic magic is insufficient for templates, where we
need to be able to refer to the specific history table to take
advantage of sqlalchemy's relationship management (2/3rds of an ORM).
So replace it with a custom made version table.

Had to change the version decorator slightly for this
2016-08-09 16:56:43 +01:00
Leo Hemsted
e88624ed4a attempt to pull logos from the admin app's static images directory
(this is configured by a config value)
2016-08-09 16:03:04 +01:00
Leo Hemsted
ebb8947290 look at service's organisation for branding to pass through to email renderer 2016-08-09 16:03:04 +01:00
Leo Hemsted
d38fdb2d11 add organisation and branding models
a service now has branding and organisation_id columns, and two new
tables have been aded to reflect these:
* branding is a static types table referring to how a service wants
  their emails to be branded:
    * 'govuk' for GOV UK branding (default)
    * 'org' for organisational branding only
    * 'both' for co-branded output with both
* organisation is a table defining an organisation's branding. this
  contains three entries, all of which are nullable
    * colour - a hex code for a coloured bar on the logo's left
    * logo - relative path for that org's logo image
    * name - the name to display on the right of the logo
2016-08-09 16:03:04 +01:00
Leo Hemsted
e631333a34 add GET /organisation and GET /organisation/id endpoints 2016-08-09 15:59:22 +01:00
Leo Hemsted
8e46cd44fd make history meta handle nullable and default better
* can now handle nullable foreign keys - would previously raise
  AttributeError
* can now handle default values - we would previously not insert
  default values that hadn't been generated yet, which if the
  field is not nullable will result in IntegrityErrors. We were
  deliberately removing 'default' attributes from columns. Only
  remove them if nullable is set to true now.
2016-08-09 15:59:22 +01:00
Leo Hemsted
a6cd490942 add organisation and branding models
a service now has branding and organisation_id columns, and two new
tables have been aded to reflect these:
* branding is a static types table referring to how a service wants
  their emails to be branded:
    * 'govuk' for GOV UK branding (default)
    * 'org' for organisational branding only
    * 'both' for co-branded output with both
* organisation is a table defining an organisation's branding. this
  contains three entries, all of which are nullable
    * colour - a hex code for a coloured bar on the logo's left
    * logo - relative path for that org's logo image
    * name - the name to display on the right of the logo
2016-08-09 15:59:16 +01:00
minglis
d67e43a6d0 Merge pull request #583 from alphagov/stats-db-updates
Stats db updates
2016-08-08 11:48:01 +01:00
Martyn Inglis
fe54fa9f73 Final pass through existing statsd endpoints to ensure they match new naming strategy.
Updates accordingly.
2016-08-08 11:23:58 +01:00
Martyn Inglis
0fa614d812 Removed unused import 2016-08-08 11:09:07 +01:00
Martyn Inglis
6b12c397a1 Ensure that configs know which environment it is.
This allows us to prefix metrics with the environment to allow stats from staging and live to go to the same statsd, and alls us to filter in the dashboard by environment.
2016-08-08 10:20:33 +01:00
Martyn Inglis
f223446f73 Refactor statsd logging
Removed all existing statsd logging and replaced with:

- statsd decorator. Infers the stat name from the decorated function call. Delegates statsd call to statsd client. Calls incr and timing for each decorated method. This is applied to all tasks and all dao methods that touch the notifications/notification_history tables

- statsd client changed to prefix all stats with "notification.api."

- Relies on https://github.com/alphagov/notifications-utils/pull/61 for request logging. Once integrated we pass the statsd client to the logger, allowing us to statsd all API calls. This passes in the start time and the method to be called (NOT the url) onto the global flask object. We then construct statsd counters and timers in the following way

	notifications.api.POST.notifications.send_notification.200

This should allow us to aggregate to the level of

	- API or ADMIN
	- POST or GET etc
	- modules
	- methods
	- status codes

Finally we count the callbacks received from 3rd parties to mapped status.
2016-08-05 10:44:43 +01:00
Leo Hemsted
143cfb526c change update_provider_stats to use billable_units
updated tests etc, and removed some old tests that are no longer relevant
2016-08-03 17:22:20 +01:00
Leo Hemsted
527a5c4eaa calculate billable units when sending an sms
don't calculate it if we're in research mode
* added tests to prove this
* removed last code referring to content_char_count
2016-08-03 16:46:05 +01:00