Commit Graph

296 Commits

Author SHA1 Message Date
Rebecca Law
331414d379 Merge pull request #2357 from alphagov/remove-edit-folder-permission
Remove the need for the edit_folders service permission.
2019-02-15 13:19:48 +00:00
Chris Hill-Scott
f679731af8 Also store consent to research against a service
It makes most sense to collect this at the same time as the estimated
volumes. Which means we need to store it somewhere; we can’t put it
straight into the ticket.
2019-02-15 10:08:22 +00:00
Chris Hill-Scott
ecdf076506 Store anticipated volumes as integers
This will make it easier to do analysis on the data. Almost all users
are submitting data in a numerical format now anyway, because we ask the
question in a sensible way.
2019-02-15 10:08:22 +00:00
Chris Hill-Scott
17e32fa5f6 Add fields to record a service’s estimated volumes
When a service go live we ask people for their estimated sending
volumes. At the moment we only put this in the ticket, and store it in
a spreadsheet.

This means that a service can
- say they want to go live
- say they are sending 100,000 emails per year
- not have created any email templates
- still see ‘create templates’ as ‘completed’ in the go live checklist

If we store this data against the service we can collect it earlier, and
then use it to determine automatically what kind of templates the user
needs to create before their go live checklist can be considered
complete.
2019-02-15 10:08:22 +00:00
Leo Hemsted
53ecaa3230 remove dvla_organisation
it's been superceeded by letter branding
2019-02-13 15:02:18 +00:00
Leo Hemsted
afc5c96927 Don't fallback to dvla_organisation if letter branding unset
The template preview app now accepts a null value for the `filename` 
parameter. If a service doesn't have a letter branding option set, 
previously we defaulted to their dvla_organisation (probably HM 
Government). Now, we pass through None, so that we generate letters 
without any logo or branding.
2019-02-13 11:58:54 +00:00
Pea Tyczynska
8ef581de69 Remove service.postage and choose_postage permission from the code and tests 2019-02-11 18:20:34 +00:00
Rebecca Law
38a599c758 Remove the need for the edit_folders service permission.
We need another PR to remove it from ServicePermissionTypes and remove the data in service_permissions.
2019-02-11 17:26:17 +00:00
Leo Hemsted
478de8bb8e fix new services being assigned to wrong letter branding
when creating a service, the api accepts a `service_domain` field that
it uses to populate the letter branding - if the service domain is
known to match an existing letter branding option, use that
automatically. However, the admin currently doesn't know about this
field yet so doesn't pass anything through - the api erroneously
searches the DB for letter branding with a domain of None - which they
currently all have.

This meant that when services were created, their letter branding was
set to the most recent row in the DB (that matched None).
2019-02-11 11:46:33 +00:00
Chris Hill-Scott
4b1336b405 Give folders to new services by default
Step 1 of 2 of turning on folders for all services.

We think it’s a feature which will be useful for the majority of
services, and we think we’ve done enough research to know that it’s
mature enough to release to all services.
2019-01-29 12:30:41 +00:00
Rebecca Law
e030c2be88 Removing platform_default as a concept. No service actually wants to send letters with the default hm-government logo so we are going to remove it as a constraint.
However, until we can create a letter without a logo, we will still default to hm-government, because the dvla_organisation is set on the service.
This does simplify the code.
Also removed the inserts to letter_branding in the data migration file, because we can deploy this before the rest of the work is finished. But we will need to do it later.
2019-01-25 15:03:01 +00:00
Rebecca Law
f11aa55e0b Adding new data models for letter branding. 2019-01-22 17:27:00 +00:00
Alexey Bezhan
1719f31909 Merge pull request #2284 from alphagov/add-count-pages-flag-to-service-notifications
Don't return pagination links for API Message log requests
2019-01-09 14:36:36 +00:00
Alexey Bezhan
47c403f6ab Don't return pagination links for API Message log requests
Flask-SQLAlchemy paginate function issues a separate query to get
the total count of rows for a given filter. This query (with
filters used by the API integration Message log page) is slow for
services with large number of notifications.

Since Message log page doesn't actually allow users to paginate
through the response (it only shows the last 50 messages) we can
use limit instead of paginate, which requires passing in another
flag from admin to the dao method.

`count` flag has been added to `paginate` in March 2018, however
there was no release of flask-sqlalchemy since then, so we need
to pull the dev version of the package from Github.
2019-01-08 13:22:27 +00:00
Rebecca Law
bd9a6352fd Optimise the query for getting the platform statistics for all services. The page should render for all time after this change.
This is one step closer to eliminating the need to read from NotificationHistory.
2019-01-04 16:45:39 +00:00
Katie Smith
1d67b55b16 Add endpoint for cancelling letters 2018-12-03 17:51:08 +00:00
Pea Tyczynska
5d806c2437 Adjust tests to new way of querying for notification stats 2018-11-06 14:40:40 +00:00
Katie Smith
e7f0a414ce Update service schema to add letter_logo_filename
Added the filename of a service's letter logo to the service schema. We want
this in the schema so that it is possible to call
`current_service.letter_logo_filename` from notifications-admin and to pass this value
through to template-preview.
2018-10-18 08:48:06 +01:00
Rebecca Law
1c0892888f Change the name of letter_class to postage, this will match the domain language used in the documentation.
Updated the tests to answer review comments.
2018-09-18 15:22:08 +01:00
Rebecca Law
79815b4f8a In order to support sending letters by first class we need a way to know how the service wants us to send the letter.
To start with this will be an attribute on the service, at the time the notification is created it will look at Service.letter_class to decide what class to use for the letter.

This PR adds Service.letter class as a nullable column.
Updated the create_service and update_service method to default the value to second.

Subsequent PRs will add the check constraint to ensure we only get first or second in the letter_class column and make that column nullable.
This can't be done all at once because it will cause an error if someone inserts or updates a service during the deploy.
2018-09-17 11:27:43 +01:00
Chris Hill-Scott
337496c5bc Deprecate service branding column
We want to drop this column. First we have to stop using it anywhere.

Needs to be made nullable so we can stop writing to it.
2018-08-30 16:05:37 +01:00
Rebecca Law
019c6a4e3a Revert "Purge notifications for configured days of retention" 2018-08-03 14:35:36 +01:00
Rebecca Law
dae29a1b61 Update the query for the notifications activity page to return the data for the days of retention if set. 2018-07-23 09:56:04 +01:00
Katie Smith
b1cfa8942a Add one_off filter when getting all notifications for a service
Added the option to filter by one_off messages to the DAO function
`get_notifications_for_service`. Previously, one-off notifications
were not returned - this has changed so that the default is for
one-off notifications to be returned. Also simplified the `include_jobs`
filter for this function.

The DAO function gets used in 3 places - for the V1 and V2 API endpoints,
which will now start to return one-off messages. It also gets used by
the admin app which needs to pass in `include_one_off=False` to the
`get_all_notifications_for_service` where we don't want one-off
notifications to show, such as the API message log page.
2018-07-18 15:08:06 +01:00
Katie Smith
0cb9e335b9 Delete old platform-stats route
We no longer need the `/platform-stats` route in the service blueprint,
because admin is using the new `/platform-stats` route in the platform stats
blueprint instead.
2018-07-10 14:59:24 +01:00
Leo Hemsted
be52adc68f add tests for monthly notification stats 2018-07-03 14:58:32 +01:00
Leo Hemsted
22496d9161 move stats endpoints to new file
(no new tests for new endpoint included in this commit)
2018-07-03 14:58:32 +01:00
Leo Hemsted
2463cd8fb5 move monthly notification stats from notification history to ft tables
it now uses the ft_notification_status table - nice and quick. if you
ask for the current tax year it'll top up the data with numbers from
the notification table.

the data is in exactly the same shape as the old endpoint - no changes
to the admin app are necessary
2018-07-03 14:56:31 +01:00
Rebecca Law
6d5eff028e Format the data for the csv. 2018-06-21 17:04:49 +01:00
Katie Smith
71796311fa Delete unused aggregate_statistics endpoint
`@service_blueprint.route('/<uuid:service_id>/fragment/aggregate_statistics')`
is not being used anywhere, so has been removed. The `provider_statistics_dao`
can also be removed, since the function this contained was only used for
the endpoint.
2018-05-21 15:03:39 +01:00
Leo Hemsted
c30f13ea67 create new stats endpoint
the admin app currently calls get_detailed_service, which gets
notification stats, adds them on to the rest of the detailed service
endpoint, and returns them. However, the admin app then only looks at
the stats - it doesn't look at the rest of the service object.

This is called in a few high profile places - the dashboard, the
notification summary page, and when you send a job. By creating a
separate endpoint that ignores the rest of the service object (and no
marshmallow too!), the hope is that we'll improve some slowness we've
been seeing.

Note: The old detailed function will still need to stay - it's used
by get_services(detailed=True) for the platform admin page.
2018-05-09 12:02:56 +01:00
Katie Smith
f2d4bc795e Add DAO function and endpoint for archiving letter contact blocks
Added a new DAO function which archives letter contact blocks by
setting archived to True. This raises an ArchiveValidationError if
trying to archive the default letter block for a service or the default
letter contact block for a template.

Added a new endpoint for archiving letter contact blocks.
2018-04-30 15:25:17 +01:00
Katie Smith
472b86f3f4 Add DAO function and endpoint for archiving SMS senders
Added a new DAO function which archives SMS senders by setting
archived to True. This raises an ArchiveValidationError, if
trying to archive a default SMS sender or an inbound number.

Added a new endpoint for archiving SMS senders.
2018-04-30 15:25:17 +01:00
Katie Smith
5f43fe23a7 Add DAO function and endpoint for archiving email reply_to addresses
Added a new DAO function which archives email reply_to addresses by
setting archived to True. This raises a new type of error, an
ArchiveValidationError, if trying to archive a default reply_to address.

Added a new endpoint for archiving email reply_to addresses.
2018-04-30 15:25:17 +01:00
Katie Smith
663021e494 Only return active SMS senders
Updated the DAO methods which return a single SMS sender and all SMS senders
to only return the non-archived senders. Changed the error raised in the Admin
interface from a SQLAlchemyError to a BadRequestError.
2018-04-30 15:25:17 +01:00
Rebecca Law
0798154fa2 Optimize the query used to return the services and todays notification totals.
By changing the created_at filter to use a specific date range I found a significant improvement to the queries performance.
The unit test needed to change because now were are returning todays date as BST the local timezone.

Query plan before

Merge Left Join  (cost=1226133.76..1226143.77 rows=1753 width=70) (actual time=5800.160..5801.657 rows=1849 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.205..2.631 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.011..1.156 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=1225980.77..1225980.99 rows=86 width=40) (actual time=5797.949..5797.984 rows=198 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=1225976.29..1225978.01 rows=86 width=40) (actual time=5797.682..5797.823 rows=198 loops=1)
->  HashAggregate  (cost=1225976.29..1225977.15 rows=86 width=48) (actual time=5797.681..5797.747 rows=198 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Seq Scan on notifications  (cost=0.00..1220610.86 rows=536543 width=48) (actual time=0.064..5482.975 rows=643799 loops=1)
Filter: (((key_type)::text <> 'TEST'::text) AND (date(created_at) = '2018-03-20'::date))
Rows Removed by Filter: 6804774
Planning time: 1.106 ms
Execution time: 5802.130 ms

Query plan after
Merge Left Join  (cost=953378.30..953388.30 rows=1753 width=70) (actual time=2380.144..2382.499 rows=1852 loops=1)
Merge Cond: (services.id = anon_1.service_id)
->  Sort  (cost=152.99..157.37 rows=1753 width=46) (actual time=2.944..3.570 rows=1762 loops=1)
Sort Key: services.id
Sort Method: quicksort  Memory: 224kB
->  Seq Scan on services  (cost=0.00..58.54 rows=1753 width=46) (actual time=0.006..1.294 rows=1762 loops=1)
Filter: active
Rows Removed by Filter: 101
->  Sort  (cost=953225.31..953225.53 rows=86 width=40) (actual time=2377.194..2377.262 rows=201 loops=1)
Sort Key: anon_1.service_id
Sort Method: quicksort  Memory: 40kB
->  Subquery Scan on anon_1  (cost=953220.83..953222.55 rows=86 width=40) (actual time=2376.797..2377.034 rows=201 loops=1)
->  HashAggregate  (cost=953220.83..953221.69 rows=86 width=48) (actual time=2376.795..2376.905 rows=201 loops=1)
Group Key: notifications.notification_type, notifications.notification_status, notifications.service_id
->  Bitmap Heap Scan on notifications  (cost=29883.14..947856.24 rows=536459 width=48) (actual time=270.061..1887.754 rows=644735 loops=1)
Recheck Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Rows Removed by Index Recheck: 947427
Filter: ((key_type)::text <> 'TEST'::text)
Heap Blocks: exact=40882 lossy=186483
->  Bitmap Index Scan on ix_notifications_created_at  (cost=0.00..29749.02 rows=536459 width=0) (actual time=258.631..258.631 rows=644849 loops=1)
Index Cond: ((created_at >= '2018-03-20 00:00:00'::timestamp without time zone) AND (created_at < '2018-03-21 00:00:00'::timestamp without time zone))
Planning time: 0.548 ms
Execution time: 2383.485 ms
2018-03-20 15:48:32 +00:00
Chris Hill-Scott
c45f0ac516 Fix code style 2018-03-09 09:53:05 +00:00
Rebecca Law
031a19d387 Added a missing test, letter search has not yet been implemented. 2018-03-09 09:53:05 +00:00
Rebecca Law
e3a75d1b7d Notification_type is a required parameter, admin app always passes it in.
Normalise for notificaiton type.
Throw InvalidRequest exception is the notification type is invalid.
2018-03-09 09:53:05 +00:00
Rebecca Law
5f25fc0db4 The template type should be passed in when doing a search by recipent.
It is possible to search for a phone number when from the email notification page and get a SMS message in return.
This also helps to optimise the query.
2018-03-09 09:53:05 +00:00
Ken Tsang
7011b90bd4 Refactor is_precompiled_letter to model 2018-03-07 23:03:03 +00:00
Ken Tsang
bca858f4a8 Return precompiled_letter flag rather than hidden 2018-03-06 12:46:06 +00:00
Ken Tsang
564504bf97 Add template hidden field in response 2018-03-06 12:46:06 +00:00
Ken Tsang
ee9b6f1fe0 Return hidden field as part of json for notification by id
- required by Admin app so that it can distinguish between created and precompiled letters
2018-03-06 12:46:06 +00:00
Leo Hemsted
aedbd37d8b don't return or accept reply_to_email in update service anymore
either
2018-02-22 11:03:35 +00:00
Leo Hemsted
703cee66f0 don't return or accept sms_sender in update service anymore
it's no longer a field on the service
2018-02-21 15:45:19 +00:00
Katie Smith
a6dae2c81b Remove unnecessary code to do with testing service name uniqueness
notifications-admin has now been changed to always pass the service_id
to the 'service/unique' endpoint. This means we don't need to cover the
case of there being no service_id and the tests can also be updated.
2018-02-15 16:32:09 +00:00
Katie Smith
e1cc8175d7 Allow services to add puntuation to or change the case of their name
Changed the '/service/unique' endpoint to optionally accept the
service_id parameter. It now doesn't matter if a user tries to change
the capitalization or add punctuation to their own service name. But
there should still be an error if a user tries to change the punctuation
or capitalization of another service.

service_id needs to be allowed to be None until notifications-admin is
updated to always pass in the service_id.
2018-02-14 10:11:46 +00:00
Ken Tsang
44f9143d08 Organisation services API endpoints 2018-02-13 14:06:43 +00:00
Leo Hemsted
ba20010f27 remove organisation from api 2018-02-07 11:39:33 +00:00