Commit Graph

2350 Commits

Author SHA1 Message Date
Rebecca Law
df2d3a95ae [WIP] 2017-11-23 14:55:49 +00:00
Alexey Bezhan
e8ce408f6a Fix an intermittent test failure when creating a template with reply_to
reply_to requires template_type to be already set, but the order
of attribute assignment is not defined when a model object is created
from a dictionary.

This adds a constructor to Template model that makes sure that
template_type is set first when multiple arguments are passed to the
constructor at once.

The problem might still exist when the template is created through the
API, so this is a temporary fix to unblock the release.
2017-11-22 16:13:45 +00:00
Alexey Bezhan
999afa7e0d Add reply_to to the list of template fields that can change 2017-11-22 14:29:37 +00:00
Alexey Bezhan
f8e1fbe3e6 Add reply_to fields to template schemas
We're hiding the `service_letter_contact_id` column since it should
only be readable and writable using the `.reply_to` wrapper.

Schemas are defined using `fields.Method` since the fields are
represented by a property on the Template model that requires
template type to be set. When creating a template, if `reply_to`
is defined using `fields.String` it gets assigned at the same time
as `template_type` (or the order of assignments is not defined),
so the schema loader attempts to set `.reply_to` on a Template
object with a `None` `template_type`, which raises an exception.

Using `fields.Method` seems to delay `.reply_to` assignment until
the Template object is created, which means it already has a
valid type.
2017-11-22 14:26:04 +00:00
Alexey Bezhan
cbce610098 Add template.service_letter_contact_id and reply_to wrapper property
Adds a relationship between Template models and service letter contact
blocks.

Depending on template type, we can have a reference to either a letter
contact block, email reply-to address or SMS sender record. This means
that in order to enforce foreign key constraints we need to define three
separate foreign key columns on the template model.

To hide this implementation detail and make it easier to access the
sender/reply-to information we define a wrapper property that returns
the value from the correct column.

The relationship and the property are only defined for letter templates
at the moment.

The setter raises an error when trying to assign a reply_to value for
non-letter templates. The exception isn't raised if the value being
assigned is `None` since it can get assigned by marshmallow schemas
and as it matches the value returned for other template types it
doesn't need to be written anywhere.
2017-11-22 14:20:53 +00:00
Alexey Bezhan
4c253bf3b9 Move common Template/TemplateHistory attributes to a base class
This allows us to avoid duplication between Template and TemplateHistory
classes and makes it easier to ensure that all columns are copied
to the TemplateHistory objects.
2017-11-22 14:15:58 +00:00
Alexey Bezhan
559639eb63 Verify that attribute exists on *History model when versioning objects
When createing a history instance of the updated object `create_history`
sets attributes using `setattr`. Since SQLAlchemy model instances are
Python objects they don't prevent new attributes being created by setattr,
which means that if history models are missing some of the columns the
attributes will still be assigned, but their values will not be persisted
by SQLAlchemy since database columns for them do not exist.

To avoid this, we check that the attribute is defined on the `history_cls`
and raise an error if it isn't.
2017-11-22 14:15:58 +00:00
Richard Chapman
a812ae1e6f Merge pull request #1412 from alphagov/rc-monthly-template-usage-endpoint
Template usage always aggregating today's stats
2017-11-22 14:05:58 +00:00
Alexey Bezhan
5e53d781e0 Add FIRETEXT_INBOUND_SMS_AUTH config variable and auth check
Checks authentication header value on inbound SMS requests from
Firetext against a list of allowed API keys set in the application
config.

At the moment, we're only logging the attempts without aborting the
requests. Once this is rolled out to production and we've checked
the logs we'll switch on the aborts and add the tests for 401 and 403
responses.
2017-11-22 09:54:42 +00:00
Richard Chapman
3ac7507f3b Template usage always aggregating today's stats
Added a check to ensure that the current date falls in between the
financial year for the year supplied by the method, so that the todays
stats will only be appended in that situation.
2017-11-21 16:29:20 +00:00
Chris Hill-Scott
22ec134668 Merge pull request #1376 from alphagov/remove-computed-sms-prefix
Remove computed SMS prefix
2017-11-21 14:29:33 +00:00
Chris Hill-Scott
ac885dd4e6 Remove computed SMS prefix
This is no longer used no that the column in the table (`prefix_sms`) is
populated.
2017-11-21 14:12:35 +00:00
Richard Chapman
f91c0a820f Fixed bug in test which was passing but not testing the services_dao
correctly

The dao_fetch_monthly_historical_usage_by_template_for_service code
wasn't being tested properly due to an bug on the test which created a
notification with the same template and hence was not testing that
a specific service would have a different template id.

- Fixed the bug in the test
- Update services_dao so that the service id check is made
2017-11-21 10:54:08 +00:00
Rebecca Law
adaadefed5 Merge pull request #1408 from alphagov/remove-unused-code
Remove unused code
2017-11-21 10:38:41 +00:00
Athanasios Voutsadakis
73dfc67400 Merge branch 'master' into bump_utils_2301 2017-11-20 17:40:45 +00:00
Rebecca Law
d5a1ec0fe3 Remove unused methods 2017-11-20 14:45:11 +00:00
Rebecca Law
577463b0ac Remove create_or_update_email_reply_to and create_or_update_letter_contact - no longer needed.
Remove Services.reply_to_email_address and Services.letter_contact_block
2017-11-20 14:33:15 +00:00
Richard Chapman
f3f8225dfe Merge pull request #1406 from alphagov/rc-monthly-template-usage-endpoint
Template usage stats updates for FY usage and bug fix
2017-11-20 11:45:30 +00:00
Athanasios Voutsadakis
a0beb190f3 Merge branch 'master' into bump_utils_2301 2017-11-20 11:16:29 +00:00
Richard Chapman
58b0658a13 Return financial year not calendar and ensure double precision values
are not returned from queries

- Updated stats_template_usage_by_month_dao.py to return the results for
financial year not calendar, as the report os for FY only and hence
only the FY data is required
- Updated services_dao.py to ensure double precision values are converted
to an int as the 'exact' function returns double precision from the
database query, as the admin code requires the value for month to be an
int
2017-11-20 10:01:45 +00:00
Venus Bailey
10f3bc24f0 Merge pull request #1404 from alphagov/vb-test-token
Inbound sms pre-implementation test 1
2017-11-20 09:30:50 +00:00
Athanasios Voutsadakis
b0546f4886 Remove unused variable 2017-11-17 16:22:36 +00:00
Chris Hill-Scott
32e19b6552 Bump utils to 23.0.0
Changes arguments to `SMSMessageTemplate` to better describe what they
do.
2017-11-17 16:20:48 +00:00
venusbb
d70fdbb5c2 no auth check 2017-11-17 15:57:41 +00:00
venusbb
12a99f9a7f Verify authorisation header being send 2017-11-17 14:36:28 +00:00
Leo Hemsted
04e3218087 ensure sample SES callbacks and SNS subscription payloads are accurate in research mode and unit tests 2017-11-17 14:34:16 +00:00
Leo Hemsted
d2154451e5 update research mode email callbacks to add process-ses-response task to queue
this involved:
* moving that task to callback_tasks to prevent circular imports
* updating the dummy research mode callbacks (with actual SNS messages from the
  ses simulator emails)
* refactoring tests
2017-11-17 13:41:45 +00:00
Leo Hemsted
0e477b7715 Merge pull request #1401 from alphagov/ses-logging
Ses logging
2017-11-17 10:42:17 +00:00
Sakis
9d2cecd001 Merge pull request #1391 from alphagov/add_proxy_header_check
Add proxy header check
2017-11-17 10:35:43 +00:00
Leo Hemsted
890604990d remove unneccessary ses endpoint tests 2017-11-17 10:31:34 +00:00
Leo Hemsted
5855adf826 removed unused HTTP ses callback handler 2017-11-16 17:53:19 +00:00
Leo Hemsted
1f317b6717 log the bounce message from SES 2017-11-16 17:46:58 +00:00
Leo Hemsted
ff449735e4 make sure fns aren't named the same 2017-11-16 17:46:41 +00:00
kentsanggds
6d36a65b65 Merge pull request #1400 from alphagov/ken-allow-include_jobs-arg-get_all_notis
Allow include jobs request param on v2 get all notifications
2017-11-16 16:41:10 +00:00
Athanasios Voutsadakis
0886f6d9f9 Merge branch 'master' into add_proxy_header_check 2017-11-16 16:36:02 +00:00
Richard Chapman
2ee91cd629 Merge pull request #1399 from alphagov/rc-monthly-template-usage-endpoint
Added a template_type to the rest call and Aggregation Bug Fix
2017-11-16 15:55:55 +00:00
Rebecca Law
35f1704a9d Merge pull request #1395 from alphagov/catch-inbound-sms-push-exception
Retry unknown exceptions for send_inbound_sms_to_service task
2017-11-16 15:47:14 +00:00
Richard Chapman
fd89a252e0 Code Style Updates
Updated the to conform with the line ends expected by the code style in
tests rather than PyCharm.
2017-11-16 15:17:27 +00:00
Richard Chapman
e7de0c0900 Added a template_type to the rest call
Added a template type which is required for the admin UI and updated the
tests. The rest tests needed updated because of the bug fix for
aggregation.
2017-11-16 15:17:27 +00:00
Richard Chapman
5ba58031b1 Fixed Bug in stats aggregation
The check for aggregation was too broad and hence was adding together
totals based on template_id and not the unqiue combination of
template id, month and year.

- Added test to test for the failure
- Added check and a test to for template_id, mon and year matches
- Celery process name did not match the task
2017-11-16 15:17:27 +00:00
Ken Tsang
0e473cace5 Update rest endpoint to handle include_jobs arg 2017-11-16 14:45:02 +00:00
Ken Tsang
df98f0dec1 Update schema to allow include_jobs on get_notis 2017-11-16 14:44:28 +00:00
Athanasios Voutsadakis
5d687d87e7 Enable header checking on preview and staging, add test 2017-11-16 12:02:09 +00:00
Athanasios Voutsadakis
819c1a2dc5 Merge branch 'master' into add_proxy_header_check 2017-11-16 11:41:50 +00:00
Richard Chapman
38e8451627 Merge pull request #1390 from alphagov/rc-monthly-template-usage-endpoint
Added new endpoint to get the new template stats
2017-11-15 16:20:04 +00:00
Richard Chapman
424f87cb68 Removed unused import
Removed an import which is no longer required.
2017-11-15 16:12:03 +00:00
Richard Chapman
17d1cfe758 Reverted Config on SQLALCHEMY_ECHO
Used for debugging not needed for production so reverted.
2017-11-15 16:00:13 +00:00
Richard Chapman
1e4850b70f Add Template name to response and filter by year
The template name should be returned for the response and the user will
pick a year, so ths adds those two features to the
notifications/templates_usage/monthly endpoint and added some tests to
test the functionality.
2017-11-15 15:55:00 +00:00
Richard Chapman
c27edf5e3c Added new endpoint to get the new template stats
Added a new endpoint which combines the usage of the stats table and the
data from the notifications tables, instead of using all the data from
the notification_history table. This should speed up the query times
and improve the page performance.

- Updated to make the stats create and update function transactional as
it actually wasn't committing the data to the table
- Added the get from the stats table
- Add a a method to combine the two results
- Added the endpoint
2017-11-15 12:58:39 +00:00
Rebecca Law
fd217e67f5 Merge branch 'master' into remove-initial-update-sms-sender 2017-11-15 12:33:38 +00:00