Commit Graph

5119 Commits

Author SHA1 Message Date
Chris Hill-Scott
54d0e64582 Merge pull request #1853 from alphagov/pass-letter-date-to-template-preview
Pass date of letter to template preview
2018-05-04 10:52:47 +01:00
Chris Hill-Scott
da43780d10 Merge pull request #1859 from alphagov/pyup-update-pytest-mock-1.9.0-to-1.10.0
Update pytest-mock to 1.10.0
2018-05-04 10:27:19 +01:00
Chris Hill-Scott
f8f1507ab1 Merge pull request #1861 from alphagov/cadw
Add letter organisation for CADW
2018-05-03 12:08:45 +01:00
Chris Hill-Scott
545127966b Add letter organisation for CADW 2018-05-03 11:50:53 +01:00
pyup-bot
c722e57657 Update pytest-mock from 1.9.0 to 1.10.0 2018-05-01 18:44:58 +01:00
Rebecca Law
8e2e94ca12 Merge pull request #1858 from alphagov/really-fix-it
Remove the truncate date function for the query.
2018-05-01 16:55:35 +01:00
Rebecca Law
a9c5ba7674 Remove the truncate date function for the query. 2018-05-01 16:48:34 +01:00
Alexey Bezhan
a2d5e77cf3 Merge pull request #1857 from alphagov/document-download-paas-manifest-env
Add document-download-api environment variables to PaaS instances
2018-05-01 15:37:36 +01:00
Alexey Bezhan
860304933c Add document-download-api environment variables to PaaS instances
Document Download keys have been added to the PaaS environment
credentials so we can use the manifest to set them on the app
instances.
2018-05-01 15:17:07 +01:00
Rebecca Law
b02a7030cc Merge pull request #1856 from alphagov/fix-date-bug
Dates are hard to get right
2018-05-01 15:00:49 +01:00
Katie Smith
6544666cd2 Merge pull request #1846 from alphagov/pyup-update-marshmallow-2.15.0-to-2.15.1
Update marshmallow to 2.15.1
2018-05-01 14:47:59 +01:00
Katie Smith
2619bc1002 Merge pull request #1843 from alphagov/pyup-update-notifications-python-client-4.8.1-to-4.8.2
Update notifications-python-client to 4.8.2
2018-05-01 14:47:48 +01:00
Rebecca Law
93c7ab6251 Because dates are confusing we need to fix this query once more.
The template statistics are returning 7 days inclusive, however the big numbers on the dashboard are for 8 days.
This PR fixes that.
2018-05-01 10:47:47 +01:00
Katie Smith
ea2fd286b5 Merge pull request #1847 from alphagov/make-reply-tos-archiveable
Allow reply-to addresses to be archived
2018-04-30 16:04:59 +01:00
Chris Hill-Scott
8ae800b117 Pass date of letter to template preview
Otherwise all letters will show the current date.

Also beefed up the tests around this part of the code a bit.
2018-04-30 15:47:49 +01:00
Katie Smith
f2d4bc795e Add DAO function and endpoint for archiving letter contact blocks
Added a new DAO function which archives letter contact blocks by
setting archived to True. This raises an ArchiveValidationError if
trying to archive the default letter block for a service or the default
letter contact block for a template.

Added a new endpoint for archiving letter contact blocks.
2018-04-30 15:25:17 +01:00
Katie Smith
472b86f3f4 Add DAO function and endpoint for archiving SMS senders
Added a new DAO function which archives SMS senders by setting
archived to True. This raises an ArchiveValidationError, if
trying to archive a default SMS sender or an inbound number.

Added a new endpoint for archiving SMS senders.
2018-04-30 15:25:17 +01:00
Katie Smith
5f43fe23a7 Add DAO function and endpoint for archiving email reply_to addresses
Added a new DAO function which archives email reply_to addresses by
setting archived to True. This raises a new type of error, an
ArchiveValidationError, if trying to archive a default reply_to address.

Added a new endpoint for archiving email reply_to addresses.
2018-04-30 15:25:17 +01:00
Katie Smith
663021e494 Only return active SMS senders
Updated the DAO methods which return a single SMS sender and all SMS senders
to only return the non-archived senders. Changed the error raised in the Admin
interface from a SQLAlchemyError to a BadRequestError.
2018-04-30 15:25:17 +01:00
Katie Smith
f810daa3c5 Only return non-archived letter contact blocks
Changed DAO functions which return one letter contact block and all
letter contact blocks for a service to only return non-archived ones.
2018-04-30 15:25:17 +01:00
Katie Smith
a57b2f50c2 Only return non-archived email reply_to addresses
Updated the DAO methods which return a single email reply_to address and
all reply_to addresses to only return the non-archived addresses.

Changed the type of error that gets raised when using the Admin
interface to be BadRequestError instead of a SQLAlchemyError.
2018-04-30 15:25:17 +01:00
Katie Smith
be28f2e2de Add 'archived' column to the 3 reply_to models
Added a new boolean column, `archived`,  with a default of False to the
three models which are used to specify the 'reply to' address for
notifications:
 * ServiceEmailReplyTo
 * ServiceSmsSender
 * ServiceLetterContact
2018-04-30 15:25:17 +01:00
Leo Hemsted
238e85ea9e Merge pull request #1833 from alphagov/read-redis
Read template usage stats from new redis keys
2018-04-30 15:15:16 +01:00
Chris Hill-Scott
55236c3173 Merge pull request #1852 from alphagov/read-job-meta-from-s3
Read job metadata from S3 metadata
2018-04-30 13:53:53 +01:00
Leo Hemsted
e35c1bde77 ensure tests have test_ prefix
also remove unused code branch

thanks, coverage!
2018-04-30 13:23:02 +01:00
Leo Hemsted
0efa223fb2 rename days_ago to midnight_n_days_ago
also add some more timezone boundary tests and minor code cleanup
2018-04-30 11:50:56 +01:00
Chris Hill-Scott
84024a9efc Refuse to process invalid files
Since the admin app won’t be checking the metadata when it starts a job
now it’s possible that someone could make a post request which attempts
to start a job for an invalid file. This commit adds a check to make
sure that can’t happen.

This is more of an extra safety thing, rather than something that the
admin app or a user will see.
2018-04-30 11:47:27 +01:00
Chris Hill-Scott
a4857c08ab Read job metadata from S3 metadata
All of our uploads now have the metadata about the job set on them in
S3. So this commit moves to using that metadata, if it’s there, instead
of the data in the body of the post request.

The aim of this is to stop the admin app having to post this data, which
means that it won’t have to keep this data in the session for the
while doing the file upload flow.
2018-04-30 11:47:13 +01:00
Chris Hill-Scott
79c6671500 Normalise whitespace and use client fixtures
Using the client fixture means that fewer nested indentation is needed.
Which, along with consistent indenting, makes the code easier to read.
2018-04-30 11:46:58 +01:00
Katie Smith
53c2bb9855 Merge pull request #1840 from alphagov/rename-is-active-columns
Change 'is_active' column names to 'archived'
2018-04-30 11:40:10 +01:00
Katie Smith
24f6ebb421 Change 'is_active' column names to 'archived'
Changed the name of the `is_active` columns to `archived` in these
tables:
* `service_email_reply_to`
* `service_sms_senders`
* `service_letter_contacts`

These columns were not being used yet, but because we describe Templates as
'archived', this keeps the names consistent.

The default value of these columns now needs to be False, not True.
2018-04-30 11:21:28 +01:00
Leo Hemsted
310b8eda4c refactor dao_get_template_usage now that limit_days no longer used
doesnt need to have any conditional stuff now - always requires a date
2018-04-30 11:13:21 +01:00
Leo Hemsted
7dc34fc3a4 set 0'd dict in redis if no data exists in redis
to get the data for a day can be reasonably slow (a few hundred
milliseconds), and if someone's viewing a service with no activity we
don't want to do that query seven times every two seconds. So if there
is no data in redis, when we get the data out of the database, we
should put it in redis so we can just grab it from there next time.

This'll happen in two cases:
* redis data is deleted
* the service sent no messages that day

additionally, make sure that we convert nicely from redis' return
values (ascii strings) to unicode keys and integer counts.
2018-04-30 11:13:21 +01:00
Leo Hemsted
85fd7c3869 add new tests for template statistics 2018-04-30 11:13:21 +01:00
Leo Hemsted
9e8b6fd00d refactor template stats endpoint to read from new redis keys
New redis keys are partitioned per service per day. New process is as
follows:

* require a count of days to filter by. Currently admin always gives 7.
* for each day, check and see if there's anything in redis. There won't
  be if either a) redis is/was down or b) the service didn't send any
  notifications that day
  - if there isn't, go to the database and get a count out.
* combine all these stats together
* get the names/template types etc out of the DB at the end.
2018-04-30 11:13:21 +01:00
Leo Hemsted
5e702449cb move days_ago to utils and make it tz aware
it's used in a few places - it should definitely know what timezones
are and return datetimes rather than dates, which are hard to work with
in terms of figuring out how tz aware they are.
2018-04-30 11:13:21 +01:00
Leo Hemsted
67019fc5a1 add day filter to template usage dao, and remove NotificationHistory
we should be very careful with when we get data from
NotificationHistory - this should probably only be from scheduled
tasks. `dao_get_template_usage` is only called from the template
statistics rest endpoint, so shouldn't ever hit template history.

also, moved tests out to new file to break up the 2k test file a bit
2018-04-30 11:13:21 +01:00
Leo Hemsted
901383777b Merge pull request #1844 from alphagov/zendesk
zendesk instead of deskpro
2018-04-27 16:59:08 +01:00
Leo Hemsted
897ab93148 zendesk instead of deskpro 2018-04-27 16:36:39 +01:00
Rebecca Law
e41834c36a Merge pull request #1848 from alphagov/update-ft-billing-for-month-and-service
Added a command to rebuild the data for a given service_id and date
2018-04-27 11:29:50 +01:00
Rebecca Law
0520dfdb60 Added a command to rebuild the data for a given service_id and date 2018-04-26 15:39:04 +01:00
Chris Waszczuk
cc6e4fabe1 Merge pull request #1839 from alphagov/make-service-stats-7-days-max
Adjust get_service_stats to get counts for a maximum of 7 days ago
2018-04-26 15:10:40 +01:00
Rebecca Law
f3dc290bcf Merge pull request #1845 from alphagov/use_ft_billing_for_usage_page
Use ft billing for usage page
2018-04-26 09:47:03 +01:00
pyup-bot
f7df64522a Update marshmallow from 2.15.0 to 2.15.1 2018-04-26 03:55:47 +01:00
Rebecca Law
51af6b27a0 Update name of method to clarify the meaning.
Update group by to use label.
Update test.
2018-04-25 17:00:19 +01:00
Rebecca Law
cff731c52d Set filter on query 2018-04-25 16:17:56 +01:00
Rebecca Law
e32debbca1 Add more tests 2018-04-25 15:40:44 +01:00
Rebecca Law
d00614205e Adding test for fetch_biling_data_for_day. 2018-04-25 14:45:51 +01:00
Rebecca Law
16ef133aa5 Refactor reporting task so that methods can be used for the annual usage page.
Still a work in progress, tests are coming.
2018-04-25 14:45:51 +01:00
Rebecca Law
bd91aac763 Write some pseudo code for ft billing 2018-04-25 14:45:51 +01:00