Commit Graph

2520 Commits

Author SHA1 Message Date
Leo Hemsted
9debc96c8c exclude non-pdfs from collate task
and fix celery kwargsg
2018-01-02 10:39:21 +00:00
Leo Hemsted
309b4d7d33 add collate-letter-pdfs task
add collate-letter-pdfs task (name pending). This retrieves a list of
letter pdf files (just the metadata, not the actual data) from s3, and
loops through them, calling the ftp task zip-and-send-letter-pdfs. It
groups them up by adding them to lists while counting the total
filesize, if it gets over a certain filesize (currently set to 500mb)
it breaks at that chunk, sends off that list of files to the ftp app,
and then starts building up a new list.

DVLA have a hard 2gb limit on how big the zip files we can send is -
however we're going to be limited by the amount of memory on the ftp
app well before we get around to handling 2gb of pdf data - so the
limit is 500mb for now. We'll adjust it after we see how ftp performs.
2018-01-02 10:39:21 +00:00
Richard Chapman
20d5a946f6 Add back in SQLALCHEMY config changes on staging
SQL Alchemy config changes were made to decrease the downtime of the
application. The last test only had 1 min of downtime in the upgrade
period i.e. 40 mins. Tested without the config changes to double
check the change had the desired effect. Adding back in so we can test
the changes under load and performance test outside of upgrade.
2017-12-22 08:21:53 +00:00
Ken Tsang
3ca97f67c9 Change live-letters-pdf to production-letters-pdf 2017-12-21 14:57:37 +00:00
Richard Chapman
66ae4ea9f2 Revert the SQLALCHEMY config changes on staging
SQL Alchemy config changes were made to decrease the downtime of the
application. The last test only had 1 min of downtime in the upgrade
period i.e. 40 mins. Reverting the changes so that the same process
can be followed to ensure the changes had the desired effect.
2017-12-21 11:31:38 +00:00
Richard Chapman
018d342408 Merge branch 'master' into rc_database_connection_settings 2017-12-20 15:59:07 +00:00
Leo Hemsted
e8967c5b05 tweak log output 2017-12-20 14:40:07 +00:00
Richard Chapman
2bc4c8ac39 Added SQLALCHEMY settings to staging for db connections
- Updated SQLALCHEMY_POOL_TIMEOUT timeout idle connections after  30 secs
- Updated SQLALCHEMY_POOL_RECYCLE to receyle the connection every 5 mins
2017-12-20 14:22:23 +00:00
Leo Hemsted
a0b87396ef change token expiry err msg for clarity 2017-12-20 13:57:34 +00:00
Leo Hemsted
9297c6d430 log user agents on api call auth.
this'll be useful for trying to figure out which services are using what
2017-12-20 13:57:34 +00:00
Leo Hemsted
687cf8526b log service id and api key id during auth
example log line:
```
API AuthError: AuthError({'token': ['Invalid token: signature, api token is not valid']}, 403, service_id=3e1ed7ea-8a05-4b4e-93ec-d7bebfea6cae, api_key_id=None)"
```
2017-12-20 13:57:34 +00:00
Leo Hemsted
e0c73256a6 log auth errors from the v2 api
(will help with tracking down client problems)
2017-12-20 13:57:12 +00:00
Rebecca Law
6fcfea7fce Merge branch 'master' of github.com:alphagov/notifications-api into populate-monthly-letter-usages 2017-12-20 09:59:30 +00:00
Rebecca Law
9051061519 Merge branch 'master' into populate-monthly-letter-usages 2017-12-19 16:55:22 +00:00
Venus Bailey
f0ea47aa05 Merge branch 'master' into vb-remove-ip-whitelist 2017-12-19 16:47:51 +00:00
Venus Bailey
56722c104c Merge branch 'master' into vb-remove-ip-whitelist 2017-12-19 15:33:28 +00:00
venusbb
a35b04f5e7 Enable MMG inbound blocking 2017-12-19 15:00:51 +00:00
Rebecca Law
a559338fe9 Merge branch 'master' into populate-monthly-letter-usages 2017-12-19 14:25:43 +00:00
Rebecca Law
cdbf34eddb Change order of query 2017-12-19 13:50:21 +00:00
Rebecca Law
e599db2684 Update the query 2017-12-19 13:47:24 +00:00
Rebecca Law
c5c40a0866 Merge branch 'master' into populate-monthly-letter-usages 2017-12-19 13:28:00 +00:00
Ken Tsang
8103540261 Renamed run-letter-pdfs to trigger-letter-pdfs-for-day
- also set optional date_to_process argument for dao_get_count_of_letters_to_process_for_date to None, so it's set in the code instead
2017-12-19 13:23:55 +00:00
Ken Tsang
3a4b8673bc Update upload_letters_pdf to use config setting for letter processing deadline 2017-12-19 13:23:55 +00:00
Ken Tsang
88f3c17463 Trigger collate-letter-pdfs-for-day task if notis to process 2017-12-19 13:23:55 +00:00
Ken Tsang
441651bbd1 Add get_count_of_letters_to_process to notifications_dao
- will get the letter notifications from day before >= letter processing deadline (17:30)
- letters_as_pdf permission is required in the service
2017-12-19 13:23:55 +00:00
Venus Bailey
6fb1e3ce96 Merge branch 'master' into vb-remove-ip-whitelist 2017-12-19 10:24:46 +00:00
Alexey Bezhan
87b56567b2 Set job notifications reply_to_text from the template.reply_to
When creating notification objects from the job sets the reply_to_text
from template's reply_to if it's present. Otherwise uses the service
default.
2017-12-19 10:23:24 +00:00
Alexey Bezhan
a98e5247b8 Update notificaiton API endpoints to use template's reply_to
Sets the reply_to_text on notification from the template value if
it's set.
2017-12-19 10:23:24 +00:00
Alexey Bezhan
3b0790f950 Add Template.get_reply_to_text helper method
Returns either template's reply_to text if set or the related field
from the default service record.

Return value can be used as default for `Notification.reply_to_text`
when per-notification value is not provided.
2017-12-19 10:23:24 +00:00
Alexey Bezhan
da247680a4 Validate that template reply_to belongs to template's service
Checks that email/sms/letter reply to object has the same service_id
as the template it's being attached to, to make sure it's not possible
to retrieve data about return addresses for other services.
2017-12-19 10:23:24 +00:00
Alexey Bezhan
509441f1d9 Add validators for service_letter_contact_id and reply_to
Validators check that service_letter_contact_id belongs to the
same service as the notification/template.

Generic reply_to validator calls the correct function for the given
type (for either notification or template). It can be used by the
template API endpoints to verify that given reply_to ID has the same
service_id as the template itself.

The original approach was to create a DB foreign key constraint,
but this caused issues with the `version_class` decorator saving
related Service objects without creating a history record.
2017-12-19 10:23:24 +00:00
Alexey Bezhan
17c239bf3d Remove accidentally committed app/__ file 2017-12-19 10:23:24 +00:00
Rebecca Law
c4d869175a Add join to LetterRates for the billing query.
Fix the yearly totals to work for letters.
2017-12-18 16:46:59 +00:00
Katie Smith
1b82afb6bb Change reply to address for invitation emails to be invitation sender
If someone receives an invitation email for Notify the reply-to address
of the email was the GOV.UK Notify email address. This has been changed
to be the email address of the user who sent the invite.

Pivotal story: https://www.pivotaltracker.com/story/show/153094646
2017-12-18 11:39:21 +00:00
Rebecca Law
ec28f48968 Merge branch 'master' into populate-monthly-letter-usages 2017-12-18 10:41:33 +00:00
Rebecca Law
995d70740a Add letter type when getting billing data for financial year 2017-12-18 10:40:13 +00:00
venusbb
568dcaa63d remove ip whitelist inbound sms codes 2017-12-18 10:25:37 +00:00
Rebecca Law
e0e64d51d5 [WIP]
Fix the query to count rather than sum the billing units.
Need to fix the query that returns the monhtly billing, there is only one row but there should be two if there are two rates.
2017-12-15 17:29:32 +00:00
Rebecca Law
c08f67ea63 Merge branch 'master' into populate-monthly-letter-usages 2017-12-15 14:26:39 +00:00
venusbb
a57264d5c8 Merge branch 'master' of https://github.com/alphagov/notifications-api into add-mmg-inbound-sms-auth 2017-12-15 12:21:24 +00:00
venusbb
ab66f5c0ac Change MMG username to look at env variable 2017-12-15 12:19:58 +00:00
Rebecca Law
9b884770e5 Remove extra if clause added when trying to merge. 2017-12-15 11:58:55 +00:00
Leo Hemsted
1ca252dcf9 put pdfs in tomorrow's dvla bucket after 17:30
So if someone sends a letter in the evening, it gets picked up the
next day
2017-12-15 11:47:52 +00:00
Rebecca Law
63799f5ff2 Merge branch 'populate-monthly-letter-usages' of github.com:alphagov/notifications-api into populate-monthly-letter-usages 2017-12-15 10:34:20 +00:00
venusbb
2e6c46c163 Merge branch 'master' of https://github.com/alphagov/notifications-api into add-mmg-inbound-sms-auth 2017-12-15 09:15:49 +00:00
Rebecca Law
012d80003a Update montly billing with letters 2017-12-14 17:17:05 +00:00
kentsanggds
5dc238671f Merge branch 'master' into ken-update-letters_pdf-billable_units 2017-12-14 16:16:00 +00:00
Ken Tsang
0ad43f0c5b Create letters pdf queue was renamed with tasks, but was lost in another merge
- needs to be correct name otherwise the delivery worker will not pick up the queue
2017-12-14 16:00:51 +00:00
Ken Tsang
0045cd6b72 Use dao_update_notification to update billable_units for letter notifications 2017-12-14 15:09:25 +00:00
Ken Tsang
f23074596b Update billable units for letters pdf task 2017-12-14 15:09:25 +00:00