Commit Graph

2612 Commits

Author SHA1 Message Date
Rebecca Law
9c99e45008 Merge pull request #1945 from alphagov/flexible-data-retention
Flexible data retention
2018-07-17 16:03:18 +01:00
Pea (Malgorzata Tyczynska)
48d1e816ef Merge pull request #1943 from alphagov/admin_platform_users_search
Add data endpoint for finding users by full or partial email
2018-07-16 15:47:23 +01:00
Pea Tyczynska
782a8ab9e7 Add data endpoint for finding users by full or partial email 2018-07-13 15:47:20 +01:00
Rebecca Law
ab695d24d5 Added new endpoints to get service data retention for a service. 2018-07-13 15:18:27 +01:00
Rebecca Law
e2a1dfeb31 New dao and endpoints to create and update service data retention. 2018-07-11 17:02:49 +01:00
Leo Hemsted
d88003d143 Merge pull request #1940 from alphagov/dashboard-day-limit
Sort out the dashboard day limits
2018-07-11 16:07:31 +01:00
Leo Hemsted
d2d267e5d8 rename limit_days to whole_days on template statistics endpoint
and return data for one more day.

we're not really limiting to 7 days - we're returning 7 entire days,
plus whatever time has elapsed since midnight today. I felt it would be
best to rename the variable to `whole_days` to imply that it's not
"limit this data set to seven days", it's "give me at least seven days".

the endpoint is backwards compatible so we can rename the variable on the front-end later
2018-07-10 15:04:35 +01:00
Katie Smith
0cb9e335b9 Delete old platform-stats route
We no longer need the `/platform-stats` route in the service blueprint,
because admin is using the new `/platform-stats` route in the platform stats
blueprint instead.
2018-07-10 14:59:24 +01:00
Katie Smith
c57f33a02d Use real letter_rates in tests
Added the letter_rate table to the list of tables which does not get
deleted after each test run and changed the tests to use the real letter
rates.

Also removed the letter rate DAO since this was only being used in
tests, so was no longer needed.
2018-07-10 11:31:18 +01:00
Leo Hemsted
7fda32b7cf dashboard functions should return data for 8 days, not 7
really, it'll be somewhere btween 7 and 8 depending on what time of day
you request it at. But if today is monday, then seven days ago is last
tuesday - but we should return data for last monday as well so that
users see a full week's worth of data

also update/clarify the tests to make sure this is being honored for
all the different widgets on the dashboard
2018-07-09 12:03:53 +01:00
Leo Hemsted
6705f9d209 Merge pull request #1933 from alphagov/delete-service-stuff
Delete service apis (callback + inbound)
2018-07-06 15:19:01 +01:00
Leo Hemsted
32415b3b14 add delete functions for inbound and callback api objects
Both service api tasks work fine if the object is unexpectedly deleted
halfway through - they both check to see if the api details are still
in the DB before trying to send the request.
2018-07-05 11:09:17 +01:00
Katie Smith
7015e240e0 Merge pull request #1929 from alphagov/paginate-complaints
Paginate complaints
2018-07-04 15:34:46 +01:00
Leo Hemsted
be52adc68f add tests for monthly notification stats 2018-07-03 14:58:32 +01:00
Leo Hemsted
00e9fee471 make sure we combine stats that are already there
eg we have yesterdays data and want to top it up with todays deltas

'use a defaultdict to make this painless'
2018-07-03 14:58:32 +01:00
Leo Hemsted
1589bb58bb make sure test keys are filtered out of stats 2018-07-03 14:58:32 +01:00
Leo Hemsted
22496d9161 move stats endpoints to new file
(no new tests for new endpoint included in this commit)
2018-07-03 14:58:32 +01:00
Leo Hemsted
089056ef72 add tests for new statistics and fact notification status functions 2018-07-03 14:58:32 +01:00
Leo Hemsted
75fdb0290f remove old monthly stats function 2018-07-03 14:57:19 +01:00
Leo Hemsted
2463cd8fb5 move monthly notification stats from notification history to ft tables
it now uses the ft_notification_status table - nice and quick. if you
ask for the current tax year it'll top up the data with numbers from
the notification table.

the data is in exactly the same shape as the old endpoint - no changes
to the admin app are necessary
2018-07-03 14:56:31 +01:00
Katie Smith
b0540098ad Fix randomly failing test
The results of the DAO function do not always get returned in the same
order, so this test was occasionally failing.
2018-07-03 11:11:49 +01:00
Katie Smith
3398e4cf93 Paginate the function to get all complaints
The DAO function to get all complaints has been paginated and the
endpoint which uses it has been updated to take a page number as a query
parameter.
2018-07-03 11:11:49 +01:00
Rebecca Law
e3b3a407c9 Merge pull request #1918 from alphagov/wrap-get_json-in-try-except
Wrap exception to return a sensible message to the client
2018-07-03 11:03:01 +01:00
Katie Smith
4228b77ebb Fix aggregate stats dao function to use start_date
Now using start_date to determine which table to query.
2018-07-03 09:26:02 +01:00
Rebecca Law
13155e24dc We are getting 500 errors when a POST notification has characters that can not be decoded or the json is malformed.
The exception is wrapped and a sensible error message is returned to the client.
2018-07-03 09:04:33 +01:00
Rebecca Law
db5663e429 Improve the performance of the query to get the last template usage.
By adding the service id, the query performance has improved greatly. It went from 6200ms to 0.04ms.
This should stop the 500s when a template is deleted.
2018-07-02 17:17:38 +01:00
Katie Smith
3bc9c06c3c Merge pull request #1919 from alphagov/new-platform-stats-endpoint
New endpoint for the updated platform admin stats page
2018-07-02 15:37:24 +01:00
Leo Hemsted
956822ee94 Merge pull request #1925 from alphagov/bump-utils-for-email-validation
Updated notifications-util to the latest version
2018-07-02 14:30:31 +01:00
Leo Hemsted
cb7f5a7166 fix letters being put in the wrong bucket when near midnight cut-off
another day, another timezone bug
2018-07-02 14:16:40 +01:00
Rebecca Law
08dcc920a2 Updated notifications-util to the latest version.
There has been an update to the email validation code.
Added an extra test for email validation.
2018-07-02 13:48:40 +01:00
Rebecca Law
c1a84fb1c4 Added tests 2018-07-02 10:59:55 +01:00
Katie Smith
b4654781be Add schema and validate platform stats requests
Added a schema to check that the start_date and end_date arguments are
both in the date format.
2018-06-29 16:14:48 +01:00
Katie Smith
cbb9f71e61 Fix bug with test notifications not showing
Test notifications are only stored in the notification table, not the
notification_history table. We were deciding which table to query for
the platform admin stats based on the start date passed to the DAO
function. This means that if the start date given was more than 7 days
ago and the end date was within the last 7 days, test notifications were
not being returned. We now use the end date to choose which table to
query which means that test notifications always get returned when they
should.
2018-06-29 16:01:45 +01:00
Katie Smith
4b030b1583 Create platform-stats blueprint
Created a platform-stats blueprint and moved the new platform stats
endpoint to the new blueprint (it was previously in the service
blueprint). Since the original platform stats route and the new platform
stats route are now in different blueprints, their view functions can
have the same name without any issues.
2018-06-29 16:01:45 +01:00
Katie Smith
5381491aae Move DAO function for new platform stats page
Moved the `fetch_new_aggregate_stats_by_date_range_for_all_services`
DAO function from the services DAO to the Notifications DAO since this
function queries the `notification` and `notification_history` tables.

Also added a test to check that the data returned from the function
takes BST into account.
2018-06-29 16:01:45 +01:00
Katie Smith
7e6076a41f Add in new endpoint with data for platform admin stats
Added in a new endpoint and DAO function to provide the data for the new
platform admin statistics page. The DAO method gets different data from
the Notifications / NotificationHistory table and also groups it differently.

The old endpoint has not been deleted yet to allow the numbers on the
old and new pages to be compared.
2018-06-29 16:01:45 +01:00
Katie Smith
7f4b828aff Add tests for get_complaint_count endpoint
* Added unit tests for the get_complaint_count endpoint
* Updated the schema to use 'date' format instead of 'datetime'
* Updated the complaint endpoint to convert start_date and end_date to
be dates instead of strings
2018-06-27 09:11:38 +01:00
Rebecca Law
28beeebbf4 Added a query and endpoint to return the count of complaints for a date-range.
Unit tests for complaints_rest.get_complaint_count have yet to be written.
Maybe there is a better name for the new endpoint.
2018-06-27 09:11:38 +01:00
Rebecca Law
6d5eff028e Format the data for the csv. 2018-06-21 17:04:49 +01:00
Rebecca Law
1ab4654396 Fix codestyle 2018-06-20 16:54:12 +01:00
Rebecca Law
709a6c38c7 Created a task to update ft_notification_status for the last three days. 2018-06-20 16:45:20 +01:00
Chris Hill-Scott
27faf7912b Search for emails or texts, depending on term
The caseworking view is going to have a page which displays emails and
text messages combined together.

In order for the search to work on this page the user needs to be able
to search for an email or a text message. This commit makes it guess
what to search for when the `notification_type` isn’t known (basically
by saying ‘if the search term is only digits they’re probably looking
searching by phone number’).
2018-06-14 10:32:04 +01:00
Rebecca Law
56c6d5101c Add some useful information to the log.
Fix unit test to actually test the right thing
2018-06-07 12:30:04 +01:00
Rebecca Law
4966da8289 Fix to handle_complaints.
The json we were getting from SES was not quite as expected, the test data now reflects what we get.
New test added, fix a test that was passing regardless.
2018-06-06 10:37:31 +01:00
Rebecca Law
c5524a3fe9 [WIP] 2018-06-05 17:23:24 +01:00
Rebecca Law
64f077f2f4 New endpoint to return data for complaints. 2018-06-05 14:25:24 +01:00
Rebecca Law
7fa42c2cc5 When SES callback is for a complaint save that to the new complaints table.
When handling the complaint we don't want to throw an exception if the message is missing fields. Only log an exception if we are unable to tie a complaint to a notification.
2018-06-04 17:29:58 +01:00
Rebecca Law
23e6b57c26 Remove email from the log message 2018-05-30 16:45:18 +01:00
Rebecca Law
e82fa2c8d4 Added a way to handle complaint responses from SES.
At this point we are just logging the message so that we can confirm the contents of the SES message.

refer to: https://www.pivotaltracker.com/story/show/157969699
2018-05-30 16:16:36 +01:00
Rebecca Law
41f3293cd5 This fixes a bug where the folder name was not correctly returning the right date because date used to compare was in UTC but the datetime used for the job start time is in BST.
Units tests have been added for the affected method.
2018-05-30 10:18:37 +01:00