Commit Graph

3193 Commits

Author SHA1 Message Date
Rebecca Law
1230495e88 Merge pull request #2213 from alphagov/check-parent-folder-same-service
Check parent folder belongs to the same service
2018-11-07 16:10:39 +00:00
Alexey Bezhan
36f41c23e1 Always use both folder and service ID when getting template folder
Currently there aren't any permission checks based on folder IDs in
the admin app or the API, so it's possible for a user to modify the
folder ID to perform operations on folders outside their service.

Our usual way to avoid this is to always use service_id filter when
fetching objects from the database.
2018-11-07 13:51:37 +00:00
Alexey Bezhan
1dbb24065d Ensure that new template folder belongs to the same service as parent
Since template folders are only linked by ID to their parent we need
to check that the parent folder belongs to the same service as the
one being created. Otherwise, admin users could modify parent ID to
create a folder outside their service.

Ideally, this check would be performed by a DB constraint, but since
parent_id can be nullable this is only possible to express using DB
triggers.

Instead, we perform the check in the API endpoint code.
2018-11-07 13:51:37 +00:00
Leo Hemsted
267c4fc07b bump requirements, fix pyflake8 things, unpin botocore/awscli 2018-11-07 13:39:08 +00:00
Pea (Malgorzata Tyczynska)
d46caa184e Merge pull request #2211 from alphagov/fix_notification_statistics_on_dashboard
Fix notification statistics on dashboard
2018-11-06 16:22:00 +00:00
Pea Tyczynska
5d806c2437 Adjust tests to new way of querying for notification stats 2018-11-06 14:40:40 +00:00
Pea Tyczynska
c146b86643 Adjust attribute key names and data types to work with format_statistics 2018-11-06 13:30:37 +00:00
Pea Tyczynska
c37c399741 Add results for notification stats for last 7 days and today 2018-11-06 11:57:08 +00:00
Rebecca Law
d15de2e753 Merge pull request #2193 from alphagov/fix-failed-statuses
Add new failed stats to statistic counts
2018-11-01 13:19:10 +00:00
Chris Hill-Scott
92dbe86fff Merge pull request #2201 from alphagov/one-off-letters-from-admin-app
Allow the admin app to send one-off letters
2018-11-01 11:40:48 +00:00
Leo Hemsted
51a7a0add7 Merge pull request #2199 from alphagov/template-folder-crud
add template folder CRUD
2018-11-01 10:44:48 +00:00
Chris Hill-Scott
674cdbd265 Allow the admin app to send one-off letters
This commit modifies the code paths the admin app uses to send one off
emails and text messages to also accept letters.

This mostly worked already, the two changes were:
- making sure that one-off letters are processed by the correct task,
  from the correct queue
- one-off letters sent from a service in research mode don’t get put on
  a queue and go straight to `delivered` (because we don’t want to send
  them for real)
2018-10-31 16:48:43 +00:00
Leo Hemsted
fbe34041d6 add template folder CRUD
* create template folder
* rename template folder
* get list of template folders for service (not nested/presented in any
  particular way)
* delete template folder

Also removed `lazy=dynamic` from the `template_folder.templates`
relationship. lazy=dynamic returns a query object (which you can then
filter further). We just want to return the entire fetched list, at
least for now.
2018-10-31 14:28:16 +00:00
Pea Tyczynska
9015ffef48 Fetch notification statistics for last 7 days frpm FactNotificationStatus 2018-10-30 16:31:57 +00:00
Rebecca Law
a76c2a0695 Merge pull request #2192 from alphagov/exclude-failed-letters-from-billing
Update billing to ignore failed letters.
2018-10-30 09:44:57 +00:00
Rebecca Law
80d02c434b Add validation-failed and virus-scan-failed in the failed counts for stats 2018-10-29 16:57:06 +00:00
Rebecca Law
8832c6d82f Update billing to ignore failed letters. 2018-10-29 14:02:40 +00:00
Leo Hemsted
7d0fa279a3 eager join folders from the Template object to keep versioning working
The `@version_class` decorator looks at every dirty (modified) model in
the session to work out which new history models to create. However, if
there are dirty items in the session, sqlalchemy might flush to the
database, clearing the whole session.

We ran into problems with the archive service function, which is
versioned for api keys, templates and services. When constructing the
TemplateHistory objects, `history_meta.py::create_history` would call
getattr on `Template.folders`, which would make a database call to join
across to the TemplateFolder objects - this would then flush the dirty
Service object from the session before the ServiceHistory object was
created.

To get around this, we eager load the Template.folder object, joining
on to it automatically when the Template is fetched. That way, it
doesn't make a SELECT mid-way through the version decorator, and the
history is preserved.

Note: This relationship is only on Template, not TemplateHistory - so
we're not doing this join every single time we send a message.
2018-10-29 11:57:24 +00:00
Leo Hemsted
0d5f8cc7df add status endpoint to '/'
paas were trying to ascertain if notify was up by looking at '/', for cert
renewal. This commit adds the status endpoint to '/', so we're not mistakenly
left for our cert to expire
2018-10-26 18:02:41 +01:00
Leo Hemsted
ac5c19c3d5 remove some pytest warnings 2018-10-26 17:54:53 +01:00
Leo Hemsted
936c8489b3 add TemplateFolder model
a TemplateFolder has a service, a name, and a parent. Parent is a
nullable foreign key pointing to another TemplateFolder instance. We
don't do any checks here for cyclical or otherwise invalid folder
structures so keep your data clean, folks!

Unsurprisingly, a Template can be part of a TemplateFolder - there's a
mapping class (template_folder_map to avoid giving it a dumb name) -
this mapping table shouldn't be interacted with directly - rather, you
should use the `Template.folder` or `TemplateFolder.templates`
relationship.
2018-10-26 17:50:31 +01:00
Katie Smith
4c630ba2e2 Make name and filename fields of DVLAOrganisation non-nullable
If either of these fields are null, previewing a letter template won't
work.
2018-10-25 13:27:44 +01:00
Katie Smith
b2c4fc2787 Add edit_folders service permission
This is a temporary permission to allow some services to see and
edit folders before the feature is turned on for everyone.
2018-10-25 11:19:36 +01:00
Rebecca Law
57de7596d8 Merge pull request #2182 from alphagov/add-more-information-to-log
Added inbound_sms_id to log output
2018-10-25 11:12:46 +01:00
Rebecca Law
b2d12e6609 Merge pull request #2171 from alphagov/update-page-count-after-antivirus-scan
Update page count after antivirus scan
2018-10-25 11:11:59 +01:00
Rebecca Law
9ee55155cc Add service id to log 2018-10-24 16:37:38 +01:00
Katie Smith
0403105d3a Merge pull request #2183 from alphagov/stop-sending-dvla-org-id
Stop passing dvla_org_id to template preview
2018-10-24 16:27:58 +01:00
Rebecca Law
4cfd5489c8 Merge branch 'master' into add-more-information-to-log 2018-10-24 16:26:44 +01:00
Rebecca Law
5459009f5a Switch order of service and inbound_sms_id in message.
Added service id to retry message
2018-10-24 16:19:17 +01:00
Rebecca Law
12d938b82b Merge branch 'master' into update-page-count-after-antivirus-scan 2018-10-24 15:13:11 +01:00
Rebecca Law
1cc2d26586 Explicitly set the billable units for update_letter_pdf_status 2018-10-24 14:50:50 +01:00
Rebecca Law
537ab2e965 Fix merge error.
Moved the billable unit calculation before the santise call.
2018-10-24 14:38:09 +01:00
Rebecca Law
00f04c33c8 Some minor refactoring.
- Updated notifications_dao.update_notification_status_by_id with an optional parameter to set the sent_by, this will eliminate a separate update to notifcaitons.
- Added the callback url to the log message, that way we can see if it's the same url failing.
- Stop sending the status callbacks for PENDING status.
2018-10-24 11:24:53 +01:00
Katie Smith
022b5b19ff Stop passing dvla_org_id to template preview
We were passing both dvla_org_id and filename to template-preview
temporarily while we switch to only using filename. Now that
template-preview is set up to use the filename, we can stop sending the
dvla_org_id too.
2018-10-23 15:52:44 +01:00
Rebecca Law
77d1bc73b5 Added inbound_sms_id to log output, the extra information may help when trying to help debug an issue. 2018-10-23 13:59:52 +01:00
Leo Hemsted
bc3fab09d0 don't log exception info for retries
it includes task args, which might contain PII. And we don't need to
know where the retry exception came from - it came from the line above
2018-10-22 11:33:16 +01:00
Alexey Bezhan
5d91ba80fb Merge pull request #2170 from alphagov/reduce-logging-for-sanitise
Reduce the error logging for sanitse method
2018-10-19 14:40:50 +01:00
Katie Smith
4dab4fa8ce Pass letter logo filename to template preview
We now pass `filename`, the filename of the letter logo to use, through
to Template Preview in addition to the `dvla_org_id`. Once Template
Preview has been updated to only use the `filename` we will stop
sending the `dvla_org_id`.
2018-10-19 10:07:11 +01:00
Katie Smith
51a18ef0b4 Merge pull request #2160 from alphagov/update-service-schema
Update service schema to add letter_logo_filename
2018-10-19 10:05:54 +01:00
Rebecca Law
02154f5fc0 Actually I decided we don't need an exception log here. 2018-10-18 16:46:04 +01:00
Rebecca Law
38c29d41a4 Reduce the error logging for sanitse method, since we ignore cysp.
Also fixed the spelling error, why not.
2018-10-18 16:43:14 +01:00
Rebecca Law
021a90f482 Merge branch 'master' into update-page-count-after-antivirus-scan 2018-10-18 16:01:59 +01:00
Leo Hemsted
e7dad9436f compare service id to UUID
it doesn't match a string 😩
I couldn't think of a good way to test this in a unit test, since
it involves changing the service id on all of the components of a
service.
2018-10-18 12:19:10 +01:00
Katie Smith
e7f0a414ce Update service schema to add letter_logo_filename
Added the filename of a service's letter logo to the service schema. We want
this in the schema so that it is possible to call
`current_service.letter_logo_filename` from notifications-admin and to pass this value
through to template-preview.
2018-10-18 08:48:06 +01:00
Leo Hemsted
c63b147a2d make sure validation_failed notifications still return pdf
it's useful to see them, and previously it was crashing.
2018-10-17 16:09:30 +01:00
Leo Hemsted
162ed0ba5f remove dupes from cfg 2018-10-17 15:29:39 +01:00
Leo Hemsted
7bf68e3664 fix failed sanitise flow
the move from virus scan to validation failed function was called with
the wrong variables, and had some internal logic that was slightly
wrong.

Also, Don't use `update_notification_by_id` for notifications if they
are not in `created`, `sending`, `pending`, or `sent`. It silently
doesn't update them. I didn't want to do a deeper investigation into
the reasons behind this terrifying state machine as part of this commit
so I just changed the functions to call `dao_update_notification`
manually
2018-10-16 17:30:39 +01:00
Leo Hemsted
9606513e1f exclude CYSP from sanitise
we know their content is already good, but they slightly exceed the
margins. Until they deploy their fixed template, lets not valiate them
2018-10-16 17:30:39 +01:00
Pea Tyczynska
e22e7245fe Use sanitised pdfs for sending and handle invalid pdfs, details below:
- pass new, sanitised pdf for sending
- move invalid pdfs to a newly created bucket
- set status fro notifications that failed pdf validation to a new status validation-failed
- adjust existing tests
2018-10-16 17:30:35 +01:00
Rebecca Law
4263117189 We were getting the page count for the letter before virus scan happened.
This PR moves setting the billale_units for the letter after virus scan has passed.
2018-10-16 15:08:15 +01:00