Commit Graph

244 Commits

Author SHA1 Message Date
Katie Smith
0cb9e335b9 Delete old platform-stats route
We no longer need the `/platform-stats` route in the service blueprint,
because admin is using the new `/platform-stats` route in the platform stats
blueprint instead.
2018-07-10 14:59:24 +01:00
Leo Hemsted
7fda32b7cf dashboard functions should return data for 8 days, not 7
really, it'll be somewhere btween 7 and 8 depending on what time of day
you request it at. But if today is monday, then seven days ago is last
tuesday - but we should return data for last monday as well so that
users see a full week's worth of data

also update/clarify the tests to make sure this is being honored for
all the different widgets on the dashboard
2018-07-09 12:03:53 +01:00
Leo Hemsted
089056ef72 add tests for new statistics and fact notification status functions 2018-07-03 14:58:32 +01:00
Leo Hemsted
75fdb0290f remove old monthly stats function 2018-07-03 14:57:19 +01:00
Katie Smith
0d4fb81235 Delete ProviderStatistics model
This is not being used anywhere, so can be deleted safely.
2018-05-21 15:03:39 +01:00
Rebecca Law
a9c5ba7674 Remove the truncate date function for the query. 2018-05-01 16:48:34 +01:00
Rebecca Law
93c7ab6251 Because dates are confusing we need to fix this query once more.
The template statistics are returning 7 days inclusive, however the big numbers on the dashboard are for 8 days.
This PR fixes that.
2018-05-01 10:47:47 +01:00
chrisw
c157de82c4 Adjust get_service_stats to get counts for a maximum of 7 days ago 2018-04-24 16:05:43 +01:00
Rebecca Law
0798154fa2 Optimize the query used to return the services and todays notification totals.
By changing the created_at filter to use a specific date range I found a significant improvement to the queries performance.
The unit test needed to change because now were are returning todays date as BST the local timezone.

Query plan before

Merge Left Join  (cost=1226133.76..1226143.77 rows=1753 width=70) (actual time=5800.160..5801.657 rows=1849 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.205..2.631 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.011..1.156 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=1225980.77..1225980.99 rows=86 width=40) (actual time=5797.949..5797.984 rows=198 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=1225976.29..1225978.01 rows=86 width=40) (actual time=5797.682..5797.823 rows=198 loops=1)
->  HashAggregate  (cost=1225976.29..1225977.15 rows=86 width=48) (actual time=5797.681..5797.747 rows=198 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Seq Scan on notifications  (cost=0.00..1220610.86 rows=536543 width=48) (actual time=0.064..5482.975 rows=643799 loops=1)
Filter: (((key_type)::text <> 'TEST'::text) AND (date(created_at) = '2018-03-20'::date))
Rows Removed by Filter: 6804774
Planning time: 1.106 ms
Execution time: 5802.130 ms

Query plan after
Merge Left Join  (cost=953378.30..953388.30 rows=1753 width=70) (actual time=2380.144..2382.499 rows=1852 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.944..3.570 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.006..1.294 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=953225.31..953225.53 rows=86 width=40) (actual time=2377.194..2377.262 rows=201 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=953220.83..953222.55 rows=86 width=40) (actual time=2376.797..2377.034 rows=201 loops=1)
->  HashAggregate  (cost=953220.83..953221.69 rows=86 width=48) (actual time=2376.795..2376.905 rows=201 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Bitmap Heap Scan on notifications  (cost=29883.14..947856.24 rows=536459 width=48) (actual time=270.061..1887.754 rows=644735 loops=1)
Recheck Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Rows Removed by Index Recheck: 947427
Filter: ((key_type)::text <> 'TEST'::text)
Heap Blocks: exact=40882 lossy=186483
->  Bitmap Index Scan on ix_notifications_created_at  (cost=0.00..29749.02 rows=536459 width=0) (actual time=258.631..258.631 rows=644849 loops=1)
Index Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Planning time: 0.548 ms
Execution time: 2383.485 ms
2018-03-20 15:48:32 +00:00
Katie Smith
db6fa457b5 Delete JobStatistics model
Also changed the delete_service_and_all_associated_db_objects function
from the services DAO to remove reference to JobStatistics table.
2018-03-12 10:48:46 +00:00
Ken Tsang
7011b90bd4 Refactor is_precompiled_letter to model 2018-03-07 23:03:03 +00:00
Ken Tsang
564504bf97 Add template hidden field in response 2018-03-06 12:46:06 +00:00
Richard Chapman
a5343fb837 Merge pull request #1628 from alphagov/rc_updated_to_use_utils_statsd
Removed statsd from the api and use the statsd in the utils library.
2018-02-07 09:07:16 +00:00
Leo Hemsted
b9446b660d exclude test keys from template usage page
they were not included in nightly task since that runs off
NotificationHistory, which doesn't include test keys. However, when you
load the page we top up the nightly stats with today's data from the
Notifications table, which *does* include test data.
2018-02-06 14:31:12 +00:00
Richard Chapman
d855b4e4ec Removed statsd from the api and use the statsd in the utils library.
The statsd code was added to the utils library a while ago, uses the
statsd from the util library and therefore consolidates the code into
once place.
2018-02-06 09:52:15 +00:00
Chris Hill-Scott
e9aba34c5e Turn letters on by default for new services
Letters is a mature enough feature now – and one that we’ve been talking
about offering for long enough – that we shouldn’t make people dig
around in the settings.

I think we’d want to wait a bit longer/indefinitely before deciding to
turn it on for existing services across the platform.
2018-01-24 13:42:11 +00:00
Leo Hemsted
4cdcc4e035 no longer add an annual billing entry creating a service
this is now handled by a separate call from the admin app
2017-12-06 14:45:44 +00:00
Leo Hemsted
b0d4044ff5 remove free_sms_fragment_limit from service
* remove from model
* still required when calling POST /service - we just call through
  from dao_create_service to add a new annual billing entry.
* removed from POST /service/<id> update_service - if you want to
  update/add a new one, use POST /service/<id>/free-sms-fragment-limit
* made sure tests create services with default 250k limit.
2017-12-06 14:45:43 +00:00
Katie Smith
f73319f5ef Add and populate crown column to services and services_history
- Added the boolean 'crown' column to services and services_history tables
- We populate this column in the same migration script by checking the
'organisation_type' of a service
2017-12-04 13:31:08 +00:00
Leo Hemsted
28088428f1 flake8 - misc flake8 errs.
* unused variables
* variables in loops overshadowing imports
* excepts with no defined exc type (tried to avoid `except Exception` too)
* history mapper is still too complex
* default variables should never be mutable
2017-11-28 14:28:01 +00:00
Leo Hemsted
28d5f9b87f flake8 - remove unused imports and ensure they're always at the top of the file 2017-11-28 14:28:01 +00:00
Richard Chapman
adfba208c4 Removed the templates/monthly endpoint
Removed the REST endpoint and the DAO that it uses as the endpoint is
no longer used by the Admin UI and the DAO is not reused anywhere
else.

- Removed REST endpoint
- Removed DAO which gets the stats
- Removed associated tests of both methods
2017-11-27 11:06:43 +00:00
Richard Chapman
3ac7507f3b Template usage always aggregating today's stats
Added a check to ensure that the current date falls in between the
financial year for the year supplied by the method, so that the todays
stats will only be appended in that situation.
2017-11-21 16:29:20 +00:00
Richard Chapman
f91c0a820f Fixed bug in test which was passing but not testing the services_dao
correctly

The dao_fetch_monthly_historical_usage_by_template_for_service code
wasn't being tested properly due to an bug on the test which created a
notification with the same template and hence was not testing that
a specific service would have a different template id.

- Fixed the bug in the test
- Update services_dao so that the service id check is made
2017-11-21 10:54:08 +00:00
Richard Chapman
58b0658a13 Return financial year not calendar and ensure double precision values
are not returned from queries

- Updated stats_template_usage_by_month_dao.py to return the results for
financial year not calendar, as the report os for FY only and hence
only the FY data is required
- Updated services_dao.py to ensure double precision values are converted
to an int as the 'exact' function returns double precision from the
database query, as the admin code requires the value for month to be an
int
2017-11-20 10:01:45 +00:00
Richard Chapman
fd89a252e0 Code Style Updates
Updated the to conform with the line ends expected by the code style in
tests rather than PyCharm.
2017-11-16 15:17:27 +00:00
Richard Chapman
e7de0c0900 Added a template_type to the rest call
Added a template type which is required for the admin UI and updated the
tests. The rest tests needed updated because of the bug fix for
aggregation.
2017-11-16 15:17:27 +00:00
Richard Chapman
5ba58031b1 Fixed Bug in stats aggregation
The check for aggregation was too broad and hence was adding together
totals based on template_id and not the unqiue combination of
template id, month and year.

- Added test to test for the failure
- Added check and a test to for template_id, mon and year matches
- Celery process name did not match the task
2017-11-16 15:17:27 +00:00
Richard Chapman
38e8451627 Merge pull request #1390 from alphagov/rc-monthly-template-usage-endpoint
Added new endpoint to get the new template stats
2017-11-15 16:20:04 +00:00
Richard Chapman
424f87cb68 Removed unused import
Removed an import which is no longer required.
2017-11-15 16:12:03 +00:00
Richard Chapman
1e4850b70f Add Template name to response and filter by year
The template name should be returned for the response and the user will
pick a year, so ths adds those two features to the
notifications/templates_usage/monthly endpoint and added some tests to
test the functionality.
2017-11-15 15:55:00 +00:00
Richard Chapman
c27edf5e3c Added new endpoint to get the new template stats
Added a new endpoint which combines the usage of the stats table and the
data from the notifications tables, instead of using all the data from
the notification_history table. This should speed up the query times
and improve the page performance.

- Updated to make the stats create and update function transactional as
it actually wasn't committing the data to the table
- Added the get from the stats table
- Add a a method to combine the two results
- Added the endpoint
2017-11-15 12:58:39 +00:00
Rebecca Law
e3f9dab884 Merge branch 'master' into remove-initial-update-sms-sender 2017-11-14 16:27:10 +00:00
Richard Chapman
b78d989d4e Updates after review
- Modified the services_dao to return an int instead of a datetime to
make usage easier and removed the BST function on year as it is not
relevant for year
- Improved tests do there is less logic by ordering the result so there
is less reliance on the template id
- Renamed variable in stats_template_usage_by_month_dao.py to make it
consistent with the method
2017-11-09 14:13:42 +00:00
Richard Chapman
ff911c30d6 Added Scheduled task to get stats for template usage
Currently some pages are timing out due to the time it takes to perform
database queries. This is an attempt to improve the performance by
performing the query against the notification history table once a day
and use the notification table for a delta between midnight and the when
the page is run and combine the results.

- Added Celery task for doing the work
- Added a dao to handle the insert and update of the stats table
- Updated tests to test the new functionality
2017-11-09 10:34:49 +00:00
Rebecca Law
0054361044 Refactor the sms sender code to remove any unused methods.
Refactor tests/db/create_service() to behave more like the real world.
Created new create_service_with_inbound_number and create_service_with_defined_sms_sender() test/db methods.
2017-11-07 14:26:18 +00:00
Rebecca Law
f5e79302cd Remove NotificationStatistics
NotificationStatistics was added as a spike but didn't work out as expected. This is finally removing all that unused code.
I'll drop the table in the next PR
2017-11-01 15:02:50 +00:00
Rebecca Law
b1b231ba90 Merge branch 'master' into sms_sender_id-for-post-notfications 2017-10-30 15:20:39 +00:00
Rebecca Law
db6668eb61 Added the notification_to_sms_sender mapping table to the purge notifications query 2017-10-30 15:17:01 +00:00
venusbb
c991df3209 Merge branch 'master' of https://github.com/alphagov/notifications-api into vb-free-sms-limit-history 2017-10-26 20:47:18 +01:00
Rebecca Law
1998034b52 Refactored the get_detailed_services to stop using marshmallow.
Also removed an extra query to services.
The query has been refactored to use an outer join to services on the notifications or notification_history table.
The expectation is that this change will improve the performance of the trial/live-services pages for platform admins.
2017-10-26 12:15:52 +01:00
venusbb
9aa7489655 incorporate reviewers comments 2017-10-26 11:49:56 +01:00
Rebecca Law
61a68a2d13 [WIP] 2017-10-25 16:47:39 +01:00
venusbb
8ad98f2806 create entry when creating a new service 2017-10-25 11:35:13 +01:00
Rebecca Law
d1561903ed [WIP]
Removing marshmallow from the get_services endpoint should give us better performance.
2017-10-25 10:40:02 +01:00
Rebecca Law
ce13e62d84 Merge pull request #1332 from alphagov/improve-platform-admin
Improve platform admin
2017-10-24 10:25:54 +01:00
Rebecca Law
bfb8528ea9 The /platform-admin takes a long time, probably because the marshmallow schema used joins to the service table to return all the service data and is inefficient.
The query itself has not been improved much at all but by not using a marshmallow schema I hope to get the performance gain I am looking for.
2017-10-23 10:58:06 +01:00
Chris Hill-Scott
46d45d8595 Make international SMS on for new services
International SMS is a mature, documented feature now. There’s no reason
it shouldn’t be available to everyone. If it’s turned off by default
then we’re relying on people finding it in the settings page to know
that it exists (which we found in research the other week that users,
who would have benefitted from having international SMS, were failing to
do).

This also fixes the problem whereby users signing up for Notify with an
international phone number (eg those working abroad for the Foreign and
Commonwealth Office) couldn’t get through the tour because they weren’t
able to send themselves the example text message (see
https://www.pivotaltracker.com/story/show/150705515).
2017-10-19 11:43:27 +01:00
venusbb
9a5addb896 Reverse codes that uses filtering of trial mode services 2017-10-02 12:00:52 +01:00
venusbb
34430ac97e tidy up codes and tests 2017-09-25 09:44:16 +01:00