Flake8 Bugbear checks for some extra things that aren’t code style
errors, but are likely to introduce bugs or unexpected behaviour. A
good example is having mutable default function arguments, which get
shared between every call to the function and therefore mutating a value
in one place can unexpectedly cause it to change in another.
This commit enables all the extra warnings provided by Flake8 Bugbear,
except for:
- the line length one (because we already lint for that separately)
- B903 Data class should either be immutable or use `__slots__` because
this seems to false-positive on some of our custom exceptions
- B902 Invalid first argument 'cls' used for instance method because
some SQLAlchemy decorators (eg `declared_attr`) make things that
aren’t formally class methods take a class not an instance as their
first argument
It disables:
- _B306: BaseException.message is removed in Python 3_ because I think
our exceptions have a custom structure that means the `.message`
attribute is still present
Matches the work done in other repos:
- https://github.com/alphagov/notifications-admin/pull/3172/files
Add different error message for email and text if content is too long.
Use utils version with is_message_too_long method implemented for email templates.
We want to add validation for an email that's too long, that way the user knows why the message is failing. At the moment if an email is too long it will get a technical failure, after the retries fail. This way the email post will get a validation error.
Once this: https://github.com/alphagov/notifications-utils/pull/804 is reverted, we can update the utils version.
SES rejects email messages bigger than 10485760 bytes (just over 10 MB per message (after base64 encoding)):
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/quotas.html#limits-message
Base64 is apparently wasteful because we use just 64 different values per byte, whereas a byte can represent
256 different characters. That is, we use bytes (which are 8-bit words) as 6-bit words. There is
a waste of 2 bits for each 8 bits of transmission data. To send three bytes of information
(3 times 8 is 24 bits), you need to use four bytes (4 times 6 is again 24 bits). Thus the base64 version
of a file is 4/3 larger than it might be. So we use 33% more storage than we could.
https://lemire.me/blog/2019/01/30/what-is-the-space-overhead-of-base64-encoding/
That brings down our max safe size to 7.5 MB == 7500000 bytes before base64 encoding
But this is not the end! The message we send to SES is structured as follows:
"Message": {
'Subject': {
'Data': subject,
},
'Body': {'Text': {'Data': body}, 'Html': {'Data': html_body}}
},
Which means that we are sending the contents of email message twice in one request: once in plain text
and once with html tags. That means our plain text content needs to be much shorter to make sure we
fit within the limit, especially since HTML body can be much byte-heavier than plain text body.
Hence, we decided to put the limit at 1MB, which is equivalent of between 250 and 500 pages of text.
That's still an extremely long email, and should be sufficient for all normal use, while at the same
time giving us safe margin while sending the emails through Amazon SES.
depending on the notification type.
Up until now, only sms messages could get message-too-long error,
but now we also need to validate the size of email messages, so
the message content needs to be tailored to the notification type.
The API needs these to check whether a service can send a notification.
This commit also updates all the tests in `test_validators.py` to take
a serialised service, not a database object.
By serialising these straight away we can:
- not go back to the database later, potentially closing the connection
sooner
- potentially cache the serialised data, meaning we don’t touch the
database at all
This commit changes the code in post notification endpoint to handle a
serialised template (ie a `dict`) rather than a database object.
This is the first step towards being able to cache the template and not
hit the database on every request.
There should be no functional changes here, it’s just refactoring.
There are some changes to the tests where the signature of functions
has changed.
Importing of the template schema has to be done at a function level,
otherwise Marshmallow gets weird.
This commit also copies the `JSONModel` class from the admin app, which
turns serialised data (a dict made from JSON) into an object on which
certain predefined properties are allowed.
This means we can still do the caching of serialised data, without
having to change too much of the code in the app, or make it ugly by
sprinkling dict lookups everywhere.
We’re not copying all of JSONModel from the admin app, just the bits we
need. We don’t need to compare or hash these objects, they’re just used
for lookups. And redefining `__getattribute__` scares Leo.
- update check_sms_content_char_count to use the SMSTemplate.is_message_too_long function, and updated the error message to align with the message returned by the admin app.
- Update the the code used by version 1 of the api to use the validate_template method.
- I did find a couple of services still using the old api, however, this change should not affect them as I checked the messages being sent and they are not too long.
- We will be sending a message to them to see if they can upgrade.
- Update the log message for authenication to include the URL - makes it easier to track if a service is using version 1 of the api.
This method has been now added to Template subclasses
used by sms, emails and letters, so we can use it to valdiate if
message is not empty.
Use new template method .is_message_empty()
Refactor function name and add a test
Admin, API and utils were all defining a value for SMS_CHAR_COUNT_LIMIT.
This value has been updated in notifications-utils to allow text
messages to be 4 fragments long and notifications-api now gets the value of
SMS_CHAR_COUNT_LIMIT from notifications-utils instead of defining it in
config.
Also updated some tests to check for the higher limit.
The whitelist was built to help developers and designers making
prototypes to do realistic usability testing of them, without having to
go through the whole go live process.
These users are sending messages using the API. The whitelist wasn’t
made available to users uploading spreadsheets. The users sending one
off messages are similar to those uploading spreadsheets, not those
using the API. Therefore they shouldn’t be able to use the whitelist to
expand the range of recipients they can send to.
Passing the argument through three methods doesn’t feel that great, but
can’t think of a better way without major refactoring…
Previously, if the SMS recipient was None there would be a 500 error
with no message displayed to the user. We now check if the recipient is
None and raise a BadRequestError if this is the case.
PR #1550 added the rate_limit column to the Service table.
This PR removes the rate limits from the config and uses rate_limit from
the Service model instead. Rate limits are still separated into 'team',
'normal' and 'test', but these values are the same for a service.
Pivotal story https://www.pivotaltracker.com/story/show/153992529
Validators check that service_letter_contact_id belongs to the
same service as the notification/template.
Generic reply_to validator calls the correct function for the given
type (for either notification or template). It can be used by the
template API endpoints to verify that given reply_to ID has the same
service_id as the template itself.
The original approach was to create a DB foreign key constraint,
but this caused issues with the `version_class` decorator saving
related Service objects without creating a history record.
International SMS is a mature, documented feature now. There’s no reason
it shouldn’t be available to everyone. If it’s turned off by default
then we’re relying on people finding it in the settings page to know
that it exists (which we found in research the other week that users,
who would have benefitted from having international SMS, were failing to
do).
This also fixes the problem whereby users signing up for Notify with an
international phone number (eg those working abroad for the Foreign and
Commonwealth Office) couldn’t get through the tour because they weren’t
able to send themselves the example text message (see
https://www.pivotaltracker.com/story/show/150705515).
Added clarification to an error message to give better debugging information.
Removed using dao_get_reply_to_by_service_id in tests to be more consistent with other code and use the test db functions or remove the need for a call altogether making the code less complex.
* Added service_email_reply_to_id to the POST /v2/notifications/email and a test to test the validator
* Caught NoResultFound exception in check_service_email_reply_to_id as it was not being caught when there there was no valid service_id or reply_to_id. Fixed failing tests which were not passing due to the NoResultFound exception and added further tests to check for the good path through the code and an test to check for an invalid service_id
* Added service_email_reply_to_id to the POST /v2/notifications/email and a test to test the validator
* Caught NoResultFound exception in check_service_email_reply_to_id as it was not being caught when there there was no valid service_id or reply_to_id. Fixed failing tests which were not passing due to the NoResultFound exception and added further tests to check for the good path through the code and an test to check for an invalid service_id
* Fixed code style in validators.py to confirm with rules
Update the name of email_reply_to_id to conform better with other attributes in the schema and the resultant code in post_notifications.py
Fixed code style in test_validators.py to confirm with rules
Added tests to test_post_notifications.py to test the email_reply_to_id being present and being incorrect, it being optional is being tested by other tests.
* Added service_email_reply_to_id to the POST /v2/notifications/email and a test to test the validator
* Added service_email_reply_to_id to the POST /v2/notifications/email and a test to test the validator
* Caught NoResultFound exception in check_service_email_reply_to_id as it was not being caught when there there was no valid service_id or reply_to_id. Fixed failing tests which were not passing due to the NoResultFound exception and added further tests to check for the good path through the code and an test to check for an invalid service_id
* Caught NoResultFound exception in check_service_email_reply_to_id as it was not being caught when there there was no valid service_id or reply_to_id. Fixed failing tests which were not passing due to the NoResultFound exception and added further tests to check for the good path through the code and an test to check for an invalid service_id
* Fixed code style in validators.py to confirm with rules
Update the name of email_reply_to_id to conform better with other attributes in the schema and the resultant code in post_notifications.py
Fixed code style in test_validators.py to confirm with rules
Added tests to test_post_notifications.py to test the email_reply_to_id being present and being incorrect, it being optional is being tested by other tests.
* Minor update after manual merge to fix check style rule break in test_validators.py where a single space was introduced.
* Updates after code review. Moved the template from the exception message as it was not required and updated the error message to match the field name in the sschema for better debugging and error identification.
* Fixed test after update of exception message
