Commit Graph

2591 Commits

Author SHA1 Message Date
kentsanggds
4773e1b51b Merge pull request #1617 from alphagov/ken-set-page-count-0-fake-dvla-response
Update the dvla response data to 0 page count
2018-02-05 10:48:28 +00:00
Rebecca Law
cf8be03c5e Fix bug with sending deskpro tickets in production.
The NOTIFY_ENVIRONMENT variable is set to `production` from the run_paas_app script, but that is overwritten with `live` in the create_app function when starting an application.
Although this is confusing and it would be good to resolve that. It is a larger piece of work. For now I have included booth strings in the if condition, that way when we do migrate the code we will not have an issue with these two methods.
2018-02-02 11:27:58 +00:00
Leo Hemsted
800fbb4c6e Merge pull request #1613 from alphagov/international-500
fix 500 when receiving delivery receipt from some international nums
2018-01-30 13:58:39 +00:00
Leo Hemsted
3d389472b8 fix 500 when receiving delivery receipt from some international nums
if the international_billing_rates.yml has `dlr: null`, that means we
don't know what delivery receipts they provide - they might not provide
any. So if we do get an update, we don't know for sure that the message
was actually delivered - lets not update it.
2018-01-30 12:08:18 +00:00
Chris Hill-Scott
aab05ac8f1 Merge pull request #1600 from alphagov/letters-for-new-services
Turn letters on by default for new services
2018-01-30 11:07:55 +00:00
Ken Tsang
9122efc45e Update the dvla response data to 0 page count
- as the response is fake, the notifications billable_unit is left at 0, the fake dvla response should also be 0. Otherwise there will be confusing logs reporting mismatched page count and billable units which are just research ones.
2018-01-29 16:39:53 +00:00
Ken Tsang
54835b56e1 Remove sent to dvla task
- task not really necessary as the status is already set to 'sending' before the task is called if it is not sending i.e. in reseach mode or using a test key
2018-01-29 14:34:22 +00:00
kentsanggds
014f738a3d Merge pull request #1609 from alphagov/ken-refactor-letter-research-process
Refactor letter research processing for letter pdfs
2018-01-26 16:39:53 +00:00
Ken Tsang
5ef9b426bb Update letter notification processing for letter pdfs in research mode
- calls create fake response file to allow functional tests to run and trigger update of status to delivered only on preview and development so that FT response files don't pollute staging and live buckets
2018-01-26 15:05:02 +00:00
Ken Tsang
377afc3ed4 Update letter processing for letter pdfs in research mode
- call create fake response file task if in research mode only on preview and development environments to not impact response files on staging and live
2018-01-26 15:04:56 +00:00
Ken Tsang
c14452d8b1 Add create fake letter response file
- used by letter functional tests to ensure that we can create the response file necessary to trigger the update to a delivered state
2018-01-26 14:45:36 +00:00
Ken Tsang
04beff7d05 Return sorted ACK files
- also fixes unit test failure due to random order of filenames
2018-01-26 14:45:36 +00:00
Ken Tsang
edd030e5dd Refactor code to move business logic
- DVLA specific logic is now moved to letters_pdf_tasks
2018-01-26 14:45:36 +00:00
Richard Chapman
49d69a84d9 Changed the time of the task to run at 00:05 as the query gets data for
the day before 0:00, so to minimise the report being out of date run the
query at 0:05.
2018-01-26 09:56:53 +00:00
Richard Chapman
eee572276f Removed the imports for statsd as the decorators are no longer being
used.
2018-01-26 09:39:11 +00:00
Richard Chapman
083bfb9ea5 Removed the performance testing statsd calls as they are no longer
needed for monitoring the performance of the v2 endpoints. They were put
in as a temporary measure whilst sustained performance testing was
taking place.
2018-01-26 09:08:45 +00:00
Ken Tsang
3355a29d01 Ensure letter personalisation is serialized correctly
The personalisation for letters can take different formats depending on
how the letter was generated, for example it can contain either
address_line_1 or addressline1. This change ensures that it is always
serialized in the same way.
2018-01-24 16:44:14 +00:00
Chris Hill-Scott
e9aba34c5e Turn letters on by default for new services
Letters is a mature enough feature now – and one that we’ve been talking
about offering for long enough – that we shouldn’t make people dig
around in the settings.

I think we’d want to wait a bit longer/indefinitely before deciding to
turn it on for existing services across the platform.
2018-01-24 13:42:11 +00:00
Venus Bailey
82e34f7699 Merge branch 'master' into letter-deskpro-ack-mod 2018-01-23 11:32:49 +00:00
venusbb
d93b0d12d1 Minor change to raising deskpro ticket and errors when letter acknowledgement fails. 2018-01-23 09:51:43 +00:00
Katie Smith
84e25d6b98 Compare letter page count with billable units in DVLA response file
We compare the page_count in the response file we receive from the DVLA
with the billable_units of the letter. If these don't match, we log an
error.
2018-01-23 08:59:01 +00:00
Katie Smith
f1c75c5c5d Change notification status of failed letters
- Changed the notification status of letters for letters that DVLA marks
as 'failed' from NOTIFICATION_TECHNICAL_FAILURE to
NOTIFICATION_TEMPORARY_FAILURE.
2018-01-23 08:59:01 +00:00
Ken Tsang
6a3c2734ca Update callback handling to process DVLA response files
- handle `RS.TXT` and `RSP.TXT` files
2018-01-23 08:59:01 +00:00
venusbb
125b810050 Merge branch 'master' of https://github.com/alphagov/notifications-api into letter_ack_send_deskpro_prod_only 2018-01-22 14:54:06 +00:00
venusbb
2edb0c0883 Grammar change to despro ticket
Grammar change to despro ticket
Strip sets of empty elements
Strip subfolder in returned zip file list
ar change to despro ticket
2018-01-22 14:51:06 +00:00
venusbb
cf30e69e8c Processing ack files only send alerts in production environment.
Processing ack files only send alerts in production environment.
Deskpro alerts include bucket names for debugging purpose.
2018-01-22 12:44:03 +00:00
Chris Hill-Scott
01cf175cb2 Don’t respect the whitelist for one off sending
The whitelist was built to help developers and designers making
prototypes to do realistic usability testing of them, without having to
go through the whole go live process.

These users are sending messages using the API. The whitelist wasn’t
made available to users uploading spreadsheets. The users sending one
off messages are similar to those uploading spreadsheets, not those
using the API. Therefore they shouldn’t be able to use the whitelist to
expand the range of recipients they can send to.

Passing the argument through three methods doesn’t feel that great, but
can’t think of a better way without major refactoring…
2018-01-22 11:42:37 +00:00
Venus Bailey
5480dba44f Merge branch 'master' into letter-S3zipchange-deskpro 2018-01-19 14:40:05 +00:00
Rebecca Law
8a126bbfbb Merge branch 'master' into raise-400-for-validation-error 2018-01-19 13:41:24 +00:00
Rebecca Law
93ce1d2503 There were some exceptions in production today where a one off message was sending a message with an invalid email address.
The admin app validation did not catch this problem. But the API did.
This PR is a small fix to catch the erorr thrown by the notifications-utils/recipient validation methods and return a 400 status rather than a 500.
This only solves the issue of the user seeing "We are experiencing technical difficulties" rather than "invalid email address"

The bug can be replicated if you enter use quotes when entering the email address.

More work needs to be done so that the admin app does the same validation as the api so the user sees a nice form validtion error rather than a 400 after clicking send.
See: https://www.pivotaltracker.com/story/show/154472625
2018-01-19 12:23:07 +00:00
venusbb
4ffb84de36 Comparison of date needs to use AWS S3 format 2018-01-19 09:24:03 +00:00
venusbb
6a790b59aa Merge branch 'master' of https://github.com/alphagov/notifications-api into letter-S3zipchange-deskpro 2018-01-18 14:45:04 +00:00
venusbb
99dda99890 Use set rather than list to compare ack file and zip files difference 2018-01-18 14:44:23 +00:00
venusbb
357ec3a7d5 Call Deskpro ticket when there is an error 2018-01-18 11:06:07 +00:00
venusbb
8f5a5f8105 Parse acknowledgement files against .ZIP.TXT created by ftp app.
- Also convert the files info to upper() for comparison rather than lower
because original file names are in upper case. The unit tests contain examples of the returned lists.
2018-01-18 10:44:36 +00:00
Alexey Bezhan
70e3b07308 Disable Deskpro letters still sending alert in preview and staging
Since preview and staging environments don't have a full DVLA
integration they're likely to contain letter notifications in
a 'sending' state. To avoid spamming Deskpro we skip the check
unless we're in a production or test environment.
2018-01-17 17:14:20 +00:00
Alexey Bezhan
29777c3123 Add a celery scheduled task to alert if letters are still sending
We should receive a response file from DVLA by 4pm the next working
day (next Monday for letters created on Friday, Saturday or Sunday).

Response file triggers a task to update the letters status from
'sending' to either 'failed' or 'delivered', at which point there
should be no letter notifications in the 'sending' state for that day.

To catch any errors in the process (eg a missing response file from
DVLA) we add a scheduled task that checks letter notifications for
previous day (or Friday when run on Monday) and raises a Deskpro
ticket if it finds any in a 'sending' state. We're checking letter
notifications based on the `sent_at` date, which is set when the
letter PDF is sent to DVLA (so for letters created after 5:30pm it
will be the next day).

The task runs at 4:30pm, which should give the response file processing
task enough time to finish if the file was uploaded at 4pm.
2018-01-17 15:35:16 +00:00
Alexey Bezhan
5298f28f80 Add utils DeskproClient and configuration variables
Deskpro client is used to create tickets from celery alerting tasks
(eg alerts for missing ack or response files from DVLA).
2018-01-17 15:04:17 +00:00
venusbb
7d84b4629a Using key query rather than try catch
Added a unit test for empty content
2018-01-17 14:45:34 +00:00
venusbb
9179802717 Fix a typo error on task argument
Modify unit test to be more robust
2018-01-17 12:21:56 +00:00
venusbb
a90596fa1b Merge branch 'master' of https://github.com/alphagov/notifications-api into raise-alert-when-no-ack-file 2018-01-16 16:20:40 +00:00
venusbb
cd2e98c388 Change datetime to use utc 2018-01-16 16:06:08 +00:00
venusbb
f273b23c25 Get ack files only from day before the ack file is received.
Take care of upper and lower case of file names and contents
Add a test for s3 get_list_of_files_by_suffix
2018-01-16 09:34:09 +00:00
Leo Hemsted
218fc5e14d only send letters in created state to ftp app for zipping
this means if we end up with some notifications sending and others not,
due to problems with the ftp connectivity for example, we don't re-send
those that worked.

As a reminder, letter pdf notifications start as created and stay that
way until we have sent the zip file to DVLA, at which point they are
updated to sending
 #
2018-01-15 17:00:00 +00:00
Katie Smith
7b4abd076b Add validation to check that sms recipient is not None
Previously, if the SMS recipient was None there would be a 500 error
with no message displayed to the user. We now check if the recipient is
None and raise a BadRequestError if this is the case.
2018-01-15 09:20:46 +00:00
venusbb
a016b3f8d6 Added a new sns callback for testing old dvla response file type.
Merge branch 'raise-alert-when-no-ack-file' of https://github.com/alphagov/notifications-api into raise-alert-when-no-ack-file
2018-01-12 16:11:06 +00:00
venusbb
24b785e7e0 Added process for dvla acknowledgement file
Daily schedule task to check ack file against zip file lists
if we haven't receive ack for a zip file, raise a 500 exception
2018-01-12 15:44:00 +00:00
venusbb
e1d150a882 Added process for dvla acknowledgement file
Daily schedule task to check ack file against zip file lists
if we haven't receive ack for a zip file, raise a 500 exception
2018-01-12 15:10:42 +00:00
Rebecca Law
9c4e43bfac Some pseudo code and notes of how to implement a check for the letter acknowledgement file. 2018-01-11 16:37:39 +00:00
Katie Smith
b07db16cd1 Get rate limit from service.rate_limit column (not config)
PR #1550 added the rate_limit column to the Service table.

This PR removes the rate limits from the config and uses rate_limit from
the Service model instead. Rate limits are still separated into 'team',
'normal' and 'test', but these values are the same for a service.

Pivotal story https://www.pivotaltracker.com/story/show/153992529
2018-01-11 10:28:11 +00:00