Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
It’s weird that `/services/<service_id>` returns `404`. The home page
for every service is the dashboard – should be possible to get there
from any other page just by stemming the URL.
Also makes it consistent with organisations, which will have
`/organisations/<org_id>`.
We pin pointed the problem to a bad loop that was calling the format_phone_number_human_readable 216 for 25 rows, yikes.
This PR fixes that performance problem.
- Updated tests and added a new mock_get_monthly_template_usage
- Deleted get_monthly_template_statistics_for_service
- Added new test to test the redirection of the old endpoint
- Removed the code for the template_history endpoint and replaced with a
redirect to the new page so that anyone is forwarded on
- Updated the template to point to the new template_usage page
Re-organised the code to re-use the months array which also was not
displaying a month where there was no stats. This now gets the months,
enumerates that array updating the templates used when there are stats
items so the users sees each month of the financial year (even if there
are no stats) when there are stats they are displayed.
- When a year contains no data ensure a default set of months is
returned so that all months can be seen in the UI
- Add the template id so the user can click through to the template
The link which when clicked allows the user to view different financial
years was pointing to the template_activity page. Updated to the link
to point to the new page.
- Implementation wasn't pythonic so updated it to be more inline with the
rest of the code for maintainability.
- The API is return a float in some cases for the month which is causing
the date to string method to fail.
The current template-activity page is slow as it is using the end point
which uses notification_history and hence is timing out. This adds a
new pages (so that they can be compared side by side) which will be
hidden until is is approved with the larger data set and tested.
Numbers over a billion overflow the two column layout. Numbers over one
hundred thousand overflow the three column layout.
This commit makes the type size smaller in these cases, so that the
numbers still fit in the boxes.
At least one of our providers gives us messages with special characters
escaped, ie a newline comes through as `\n`, not a literal newline. We
shouldn’t be showing these backslashes to any of our users.
Python has built in codecs for dealing with encoding/decoding of
strings – see
https://docs.python.org/3/library/codecs.html#text-encodings
for details. Using these builtins is safer than trying to do anything
regex or parsing-based.
The GDS Way™[1] recommends using Flake8 to lint Python projects.
This commit takes the Flake8 config from Digital Marketplace API[2] and
removes the bits we don’t need.
It changes the `max_complexity` setting to 14, which is the most complex
code we have in this repo currently (we shouldn’t be writing code _more_
complex than what we already have).
This commit also fixes the errors found by Flake8, which includes 6(!)
tests which were never getting run because they had the same names as
existing tests.
Here is a full list of the errors that were found and fixed:
```
./app/__init__.py:2:1: F401 're' imported but unused
./app/__init__.py:4:1: F401 'json' imported but unused
./app/__init__.py:8:1: F401 'dateutil' imported but unused
./app/__init__.py:11:1: F401 'flask.escape' imported but unused
./app/__init__.py:41:1: F401 'app.proxy_fix' imported but unused
./app/__init__.py:129:5: F821 undefined name 'proxy_fix'
./app/__init__.py:221:19: F821 undefined name 'highlight'
./app/__init__.py:221:35: F821 undefined name 'JavascriptLexer'
./app/__init__.py:221:54: F821 undefined name 'HtmlFormatter'
./app/config.py:2:1: F401 'datetime.timedelta' imported but unused
./app/event_handlers.py:2:1: F401 'flask_login.current_user' imported but unused
./app/utils.py:11:1: F401 'dateutil.parser' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.two_factor' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.notifications' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.add_service' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.forgot_password' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.inbound_number' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.styleguide' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.organisations' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.letter_jobs' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.verify' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.conversation' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.api_keys' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.send' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.dashboard' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.jobs' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.manage_users' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.sign_in' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.sign_out' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.code_not_received' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.invites' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.platform_admin' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.providers' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.service_settings' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.index' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.new_password' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.user_profile' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.feedback' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.choose_service' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.templates' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.register' imported but unused
./app/main/forms.py:12:1: F401 'wtforms.SelectField' imported but unused
./app/main/views/api_keys.py:37:29: E241 multiple spaces after ':'
./app/main/views/feedback.py:3:1: F401 'flask.flash' imported but unused
./app/main/views/feedback.py:122:17: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/inbound_number.py:1:1: F401 'flask.url_for' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.session' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.redirect' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.request' imported but unused
./app/main/views/inbound_number.py:13:1: F401 'flask.jsonify' imported but unused
./app/main/views/jobs.py:31:1: F401 'app.utils.get_template' imported but unused
./app/main/views/letter_jobs.py:1:1: F401 'datetime' imported but unused
./app/main/views/letter_jobs.py:6:1: F401 'app.format_datetime_24h' imported but unused
./app/main/views/manage_users.py:111:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/notifications.py:121:5: F841 local variable 'status_args' is assigned to but never used
./app/main/views/organisations.py:1:1: F401 'flask.request' imported but unused
./app/main/views/service_settings.py:77:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/service_settings.py:82:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/service_settings.py:420:13: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/sign_in.py:12:1: F401 'flask_login.confirm_login' imported but unused
./app/main/views/sign_in.py:17:1: F401 'app.service_api_client' imported but unused
./app/main/views/sign_in.py:62:13: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/templates.py:4:1: F401 'flask.json' imported but unused
./app/main/views/templates.py:17:1: F401 'notifications_utils.formatters.escape_html' imported but unused
./app/main/views/templates.py:23:1: F401 'app.utils.get_help_argument' imported but unused
./app/main/views/templates.py:64:13: E123 closing bracket does not match indentation of opening bracket's line
./app/notify_client/service_api_client.py:6:1: F401 '.notification_api_client' imported but unused
./app/notify_client/user_api_client.py:1:1: F401 'uuid' imported but unused
./app/notify_client/user_api_client.py:3:1: F401 'flask.session' imported but unused
./tests/__init__.py:1:1: F401 'csv' imported but unused
./tests/app/main/test_asset_fingerprinter.py:2:1: F401 'os' imported but unused
./tests/app/main/test_asset_fingerprinter.py:4:1: F401 'unittest.mock' imported but unused
./tests/app/main/test_asset_fingerprinter.py:98:9: F841 local variable 'string_with_unicode_character' is assigned to but never used
./tests/app/main/test_errorhandlers.py:2:1: F401 'flask.url_for' imported but unused
./tests/app/main/test_permissions.py:26:13: F841 local variable 'response' is assigned to but never used
./tests/app/main/test_placeholder_form.py:3:1: F401 'wtforms.Label' imported but unused
./tests/app/main/test_placeholder_form.py:11:10: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:10:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:23:65: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:37:48: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:51:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:65:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/views/test_accept_invite.py:356:5: F841 local variable 'element' is assigned to but never used
./tests/app/main/views/test_activity.py:11:1: F811 redefinition of unused 'mock_get_notifications' from line 11
./tests/app/main/views/test_activity.py:18:1: F401 'datetime.datetime' imported but unused
./tests/app/main/views/test_activity.py:102:5: F841 local variable 'content' is assigned to but never used
./tests/app/main/views/test_activity.py:104:5: F841 local variable 'notification' is assigned to but never used
./tests/app/main/views/test_activity.py:337:5: F841 local variable '_notifications_mock' is assigned to but never used
./tests/app/main/views/test_activity.py:373:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_activity.py:378:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_activity.py:404:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_activity.py:407:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_api_keys.py:354:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_conversation.py:5:1: F401 'bs4.BeautifulSoup' imported but unused
./tests/app/main/views/test_conversation.py:198:5: F841 local variable 'mock_get_inbound_sms' is assigned to but never used
./tests/app/main/views/test_dashboard.py:53:5: F841 local variable 'mock_template_stats' is assigned to but never used
./tests/app/main/views/test_dashboard.py:72:5: F841 local variable 'mock_template_stats' is assigned to but never used
./tests/app/main/views/test_jobs.py:2:1: F401 'uuid' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.urlparse' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.quote' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.parse_qs' imported but unused
./tests/app/main/views/test_jobs.py:9:1: F401 'app.main.views.jobs.get_status_filters' imported but unused
./tests/app/main/views/test_jobs.py:10:1: F401 'tests.notification_json' imported but unused
./tests/app/main/views/test_letters.py:6:1: F401 'tests.service_json' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.REQUESTED_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.DELIVERED_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.SENDING_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.FAILURE_STATUSES' imported but unused
./tests/app/main/views/test_platform_admin.py:242:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_platform_admin.py:247:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:3:1: F401 'unittest.mock.Mock' imported but unused
./tests/app/main/views/test_send.py:18:1: F811 redefinition of unused 'mock_get_service' from line 18
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.multiple_letter_contact_blocks' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.no_sms_senders' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.multiple_sms_senders' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.no_letter_contact_blocks' imported but unused
./tests/app/main/views/test_send.py:102:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_send.py:870:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_send.py:1367:5: F841 local variable 'service_id' is assigned to but never used
./tests/app/main/views/test_send.py:1451:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:1620:80: E226 missing whitespace around arithmetic operator
./tests/app/main/views/test_send.py:1909:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:1912:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_service_settings.py:13:1: F811 redefinition of unused 'no_reply_to_email_addresses' from line 13
./tests/app/main/views/test_service_settings.py:13:1: F401 'tests.conftest.single_reply_to_email_address' imported but unused
./tests/app/main/views/test_service_settings.py:28:5: E123 closing bracket does not match indentation of opening bracket's line
./tests/app/main/views/test_service_settings.py:104:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:166:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:186:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:217:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:237:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:257:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:307:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:340:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:466:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:555:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:615:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:719:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:874:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:902:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:954:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:986:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1101:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1121:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1271:1: F811 redefinition of unused 'test_set_letter_contact_block_saves' from line 1189
./tests/app/main/views/test_service_settings.py:1433:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1495:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1540:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1570:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1589:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1621:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1641:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1658:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1676:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1697:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1759:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1775:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_templates.py:3:1: F401 'uuid' imported but unused
./tests/app/main/views/test_templates.py:11:1: F401 'tests.conftest.mock_get_user' imported but unused
./tests/app/main/views/test_templates.py:514:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:672:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:795:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:835:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_two_factor.py:67:13: E126 continuation line over-indented for hanging indent
./tests/app/notify_client/test_notification_client.py:79:5: F841 local variable 'mock_post' is assigned to but never used
```
1. https://gds-way.cloudapps.digital/manuals/programming-languages/python/linting.html#how-to-use-flake8
2. d5ab8afef4/.flake8
In user research, we’ve seen users copy/pasting the contents of the
inbound SMS page into a spreadsheet, in order to keep a record of the
messages they receive. They even went as far as to write a macro which
fixed the errors caused by copying and pasting.
It would be much easier if we just gave them the data already in a
spreadsheet format. Which is what this commit does.
One caveat is that, because spreadsheets can contain executable code (ie
formulas), and because we’re populating the spreadsheet with
user-submitted data (albeit via SMS) we need to be careful about
injection attacks.
The details of how these attacks work are detailed here (interesting
reading): http://georgemauer.net/2017/10/07/csv-injection.html
The mitigation is to not allow characters which initialise a formula
at the start of the cell.
Only services that have inbound SMS turned on should be able to see the
dashboard and ‘Received messages’ page.
There’s probably a cleaner way (decorator) of doing this permissions
stuff, but I think it can wait until we ship this.
This commit adds two things:
a section on the dashboard to show how many inbound messages the
service has received in the last 7 days, and how recently an inbound
message has been received
---
Doesn’t show the contents of any messages, just like how the rest of the
dashboard is an aggregation, never individual messages.
a page to show all the inbound messages the service has received in
the last 7 days
---
This shows the first line of the message. Eventually this will link
through to a ‘conversation’ page, where a service can see all the
messages it’s received from a given phone number.
