Reject CSV / Spreadsheet files larger than 10Mb

This is a quick additional check to protect the user:

- From getting a CloudFront 502 error if the file takes too
long to upload. I was surprised to find it takes about 1 minute
to upload a 70Mb file to S3.*

- From getting a CloudFront 502 error when we follow the redirect
and run through the slow processing code in utils that builds a
RecipientCSV [1].

For context, a CSV with 100K rows and a few columns is around 5Mb,
so a 10Mb limit should be enough. Analysis over the past week shows
that the vast majority of CSV uploads are actually < 2.5Mb.

I haven't added any tests for this because:

- The check isn't critical, as the worst case scenario is the user
gets a worse error than this in-app one.

- There's no easy way to mock the validation, and I didn't want to
have a test that depends on a 10Mb+ file.

*We're using "key.put" to upload the file, when we could be doing
a multipart upload [2]. However, I tried this myself with a chunk
size of 1000 bytes and found it only led to a marginal improvement.

[1]: https://github.com/alphagov/notifications-utils/pull/930
[2]: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html
This commit is contained in:
Ben Thorner
2021-12-07 12:39:37 +00:00
parent 787cb3ef1f
commit 0ce7f72b07
2 changed files with 29 additions and 2 deletions

View File

@@ -9,6 +9,7 @@ from flask_login import current_user
from flask_wtf import FlaskForm as Form
from flask_wtf.file import FileAllowed
from flask_wtf.file import FileField as FileField_wtf
from flask_wtf.file import FileSize
from notifications_utils.columns import Columns
from notifications_utils.countries.data import Postage
from notifications_utils.formatters import strip_all_whitespace
@@ -1512,8 +1513,14 @@ class ChangePasswordForm(StripWhitespaceForm):
class CsvUploadForm(StripWhitespaceForm):
file = FileField('Add recipients', validators=[DataRequired(
message='Please pick a file'), CsvFileValidator()])
file = FileField('Add recipients', validators=[
DataRequired(message='Please pick a file'),
CsvFileValidator(),
FileSize(
max_size=10e6, # 10Mb
message='File must be smaller than 10Mb'
)
])
class ChangeNameForm(StripWhitespaceForm):