mirror of
https://github.com/GSA/notifications-admin.git
synced 2026-08-15 07:18:58 -04:00
Reject CSV / Spreadsheet files larger than 10Mb
This is a quick additional check to protect the user: - From getting a CloudFront 502 error if the file takes too long to upload. I was surprised to find it takes about 1 minute to upload a 70Mb file to S3.* - From getting a CloudFront 502 error when we follow the redirect and run through the slow processing code in utils that builds a RecipientCSV [1]. For context, a CSV with 100K rows and a few columns is around 5Mb, so a 10Mb limit should be enough. Analysis over the past week shows that the vast majority of CSV uploads are actually < 2.5Mb. I haven't added any tests for this because: - The check isn't critical, as the worst case scenario is the user gets a worse error than this in-app one. - There's no easy way to mock the validation, and I didn't want to have a test that depends on a 10Mb+ file. *We're using "key.put" to upload the file, when we could be doing a multipart upload [2]. However, I tried this myself with a chunk size of 1000 bytes and found it only led to a marginal improvement. [1]: https://github.com/alphagov/notifications-utils/pull/930 [2]: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html
This commit is contained in:
@@ -9,6 +9,7 @@ from flask_login import current_user
|
||||
from flask_wtf import FlaskForm as Form
|
||||
from flask_wtf.file import FileAllowed
|
||||
from flask_wtf.file import FileField as FileField_wtf
|
||||
from flask_wtf.file import FileSize
|
||||
from notifications_utils.columns import Columns
|
||||
from notifications_utils.countries.data import Postage
|
||||
from notifications_utils.formatters import strip_all_whitespace
|
||||
@@ -1512,8 +1513,14 @@ class ChangePasswordForm(StripWhitespaceForm):
|
||||
|
||||
|
||||
class CsvUploadForm(StripWhitespaceForm):
|
||||
file = FileField('Add recipients', validators=[DataRequired(
|
||||
message='Please pick a file'), CsvFileValidator()])
|
||||
file = FileField('Add recipients', validators=[
|
||||
DataRequired(message='Please pick a file'),
|
||||
CsvFileValidator(),
|
||||
FileSize(
|
||||
max_size=10e6, # 10Mb
|
||||
message='File must be smaller than 10Mb'
|
||||
)
|
||||
])
|
||||
|
||||
|
||||
class ChangeNameForm(StripWhitespaceForm):
|
||||
|
||||
Reference in New Issue
Block a user