The new API response for template statistics returns separate
count for each status. We get rid of template stats for cancelled
notifications and group the rest of the statuses together.
At the moment the dashboard does two API calls to find out if a service
has:
1. Scheduled jobs
2. Normal jobs
API calls are slow because they are synchronous, go over the network and
touch the database. We can’t cache these API calls because:
- a scheduled job could become a normal job at any time
- the statistics on a normal job are constantly updating
However there are plenty of services which don’t have any jobs, and
probably never will. And finding out if a service has any jobs is
reliably cacheable (because as soon as a service creates its first job
it has some jobs).
So this commit:
- refactors the way we get scheduled/normal jobs into the job_api_client
to make the view a bit slimmer
- makes an additional, Redis-wrapped call to find out if any jobs exist
before trying to get the jobs
This should result in a speedup on the dashboard, and can be used in the
future if there’s anywhere else we want to show or hide something
depending on whether a service has created any jobs (I have some ideas).
we're not actually looking at the detailed service aspects - just
the stats. We're doing this in three places:
* dashboard
* notification activity page
* when checking jobs to see if we're over the daily limit
change these places to use a new api endpoint (service/id/statistics),
which hopefully be a little more performant, and will definitely be a
little more organised - moving away from generic endpoints with loads
of optional parameters.
We still need the detailed endpoints for the platform admin page tho.
Depends on https://github.com/alphagov/notifications-api/pull/1865
we were seeing isort produce different outputs locally and in docker -
this was due to it having different opinions about whether the tests
module (ie all our unit tests) is a first party (local) or third party
(pip installed) import. It's a first party import, so by defining this
in the setup.cfg isort settings, we can force it to be consistent
between environments.
Note: I don't know why it was different in the first place though
Done using isort[1], with the following command:
```
isort -rc ./app ./tests
```
Adds linting to the `run_tests.sh` script to stop badly-sorted imports
getting re-introduced.
Chosen style is ‘Vertical Hanging Indent’ with trailing commas, because
I think it gives the cleanest diffs, eg:
```
from third_party import (
lib1,
lib2,
lib3,
lib4,
)
```
1. https://pypi.python.org/pypi/isort
It’s weird that `/services/<service_id>` returns `404`. The home page
for every service is the dashboard – should be possible to get there
from any other page just by stemming the URL.
Also makes it consistent with organisations, which will have
`/organisations/<org_id>`.
- Updated tests and added a new mock_get_monthly_template_usage
- Deleted get_monthly_template_statistics_for_service
- Added new test to test the redirection of the old endpoint
Numbers over a billion overflow the two column layout. Numbers over one
hundred thousand overflow the three column layout.
This commit makes the type size smaller in these cases, so that the
numbers still fit in the boxes.
At least one of our providers gives us messages with special characters
escaped, ie a newline comes through as `\n`, not a literal newline. We
shouldn’t be showing these backslashes to any of our users.
Python has built in codecs for dealing with encoding/decoding of
strings – see
https://docs.python.org/3/library/codecs.html#text-encodings
for details. Using these builtins is safer than trying to do anything
regex or parsing-based.
The GDS Way™[1] recommends using Flake8 to lint Python projects.
This commit takes the Flake8 config from Digital Marketplace API[2] and
removes the bits we don’t need.
It changes the `max_complexity` setting to 14, which is the most complex
code we have in this repo currently (we shouldn’t be writing code _more_
complex than what we already have).
This commit also fixes the errors found by Flake8, which includes 6(!)
tests which were never getting run because they had the same names as
existing tests.
Here is a full list of the errors that were found and fixed:
```
./app/__init__.py:2:1: F401 're' imported but unused
./app/__init__.py:4:1: F401 'json' imported but unused
./app/__init__.py:8:1: F401 'dateutil' imported but unused
./app/__init__.py:11:1: F401 'flask.escape' imported but unused
./app/__init__.py:41:1: F401 'app.proxy_fix' imported but unused
./app/__init__.py:129:5: F821 undefined name 'proxy_fix'
./app/__init__.py:221:19: F821 undefined name 'highlight'
./app/__init__.py:221:35: F821 undefined name 'JavascriptLexer'
./app/__init__.py:221:54: F821 undefined name 'HtmlFormatter'
./app/config.py:2:1: F401 'datetime.timedelta' imported but unused
./app/event_handlers.py:2:1: F401 'flask_login.current_user' imported but unused
./app/utils.py:11:1: F401 'dateutil.parser' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.two_factor' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.notifications' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.add_service' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.forgot_password' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.inbound_number' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.styleguide' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.organisations' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.letter_jobs' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.verify' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.conversation' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.api_keys' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.send' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.dashboard' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.jobs' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.manage_users' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.sign_in' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.sign_out' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.code_not_received' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.invites' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.platform_admin' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.providers' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.service_settings' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.index' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.new_password' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.user_profile' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.feedback' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.choose_service' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.templates' imported but unused
./app/main/__init__.py:5:1: F401 'app.main.views.register' imported but unused
./app/main/forms.py:12:1: F401 'wtforms.SelectField' imported but unused
./app/main/views/api_keys.py:37:29: E241 multiple spaces after ':'
./app/main/views/feedback.py:3:1: F401 'flask.flash' imported but unused
./app/main/views/feedback.py:122:17: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/inbound_number.py:1:1: F401 'flask.url_for' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.session' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.redirect' imported but unused
./app/main/views/inbound_number.py:1:1: F401 'flask.request' imported but unused
./app/main/views/inbound_number.py:13:1: F401 'flask.jsonify' imported but unused
./app/main/views/jobs.py:31:1: F401 'app.utils.get_template' imported but unused
./app/main/views/letter_jobs.py:1:1: F401 'datetime' imported but unused
./app/main/views/letter_jobs.py:6:1: F401 'app.format_datetime_24h' imported but unused
./app/main/views/manage_users.py:111:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/notifications.py:121:5: F841 local variable 'status_args' is assigned to but never used
./app/main/views/organisations.py:1:1: F401 'flask.request' imported but unused
./app/main/views/service_settings.py:77:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/service_settings.py:82:9: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/service_settings.py:420:13: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/sign_in.py:12:1: F401 'flask_login.confirm_login' imported but unused
./app/main/views/sign_in.py:17:1: F401 'app.service_api_client' imported but unused
./app/main/views/sign_in.py:62:13: E123 closing bracket does not match indentation of opening bracket's line
./app/main/views/templates.py:4:1: F401 'flask.json' imported but unused
./app/main/views/templates.py:17:1: F401 'notifications_utils.formatters.escape_html' imported but unused
./app/main/views/templates.py:23:1: F401 'app.utils.get_help_argument' imported but unused
./app/main/views/templates.py:64:13: E123 closing bracket does not match indentation of opening bracket's line
./app/notify_client/service_api_client.py:6:1: F401 '.notification_api_client' imported but unused
./app/notify_client/user_api_client.py:1:1: F401 'uuid' imported but unused
./app/notify_client/user_api_client.py:3:1: F401 'flask.session' imported but unused
./tests/__init__.py:1:1: F401 'csv' imported but unused
./tests/app/main/test_asset_fingerprinter.py:2:1: F401 'os' imported but unused
./tests/app/main/test_asset_fingerprinter.py:4:1: F401 'unittest.mock' imported but unused
./tests/app/main/test_asset_fingerprinter.py:98:9: F841 local variable 'string_with_unicode_character' is assigned to but never used
./tests/app/main/test_errorhandlers.py:2:1: F401 'flask.url_for' imported but unused
./tests/app/main/test_permissions.py:26:13: F841 local variable 'response' is assigned to but never used
./tests/app/main/test_placeholder_form.py:3:1: F401 'wtforms.Label' imported but unused
./tests/app/main/test_placeholder_form.py:11:10: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:10:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:23:65: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:37:48: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:51:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/test_two_factor_form.py:65:67: F841 local variable 'req' is assigned to but never used
./tests/app/main/views/test_accept_invite.py:356:5: F841 local variable 'element' is assigned to but never used
./tests/app/main/views/test_activity.py:11:1: F811 redefinition of unused 'mock_get_notifications' from line 11
./tests/app/main/views/test_activity.py:18:1: F401 'datetime.datetime' imported but unused
./tests/app/main/views/test_activity.py:102:5: F841 local variable 'content' is assigned to but never used
./tests/app/main/views/test_activity.py:104:5: F841 local variable 'notification' is assigned to but never used
./tests/app/main/views/test_activity.py:337:5: F841 local variable '_notifications_mock' is assigned to but never used
./tests/app/main/views/test_activity.py:373:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_activity.py:378:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_activity.py:404:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_activity.py:407:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_api_keys.py:354:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_conversation.py:5:1: F401 'bs4.BeautifulSoup' imported but unused
./tests/app/main/views/test_conversation.py:198:5: F841 local variable 'mock_get_inbound_sms' is assigned to but never used
./tests/app/main/views/test_dashboard.py:53:5: F841 local variable 'mock_template_stats' is assigned to but never used
./tests/app/main/views/test_dashboard.py:72:5: F841 local variable 'mock_template_stats' is assigned to but never used
./tests/app/main/views/test_jobs.py:2:1: F401 'uuid' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.urlparse' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.quote' imported but unused
./tests/app/main/views/test_jobs.py:3:1: F401 'urllib.parse.parse_qs' imported but unused
./tests/app/main/views/test_jobs.py:9:1: F401 'app.main.views.jobs.get_status_filters' imported but unused
./tests/app/main/views/test_jobs.py:10:1: F401 'tests.notification_json' imported but unused
./tests/app/main/views/test_letters.py:6:1: F401 'tests.service_json' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.REQUESTED_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.DELIVERED_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.SENDING_STATUSES' imported but unused
./tests/app/main/views/test_notifications.py:5:1: F401 'app.utils.FAILURE_STATUSES' imported but unused
./tests/app/main/views/test_platform_admin.py:242:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_platform_admin.py:247:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:3:1: F401 'unittest.mock.Mock' imported but unused
./tests/app/main/views/test_send.py:18:1: F811 redefinition of unused 'mock_get_service' from line 18
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.multiple_letter_contact_blocks' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.no_sms_senders' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.multiple_sms_senders' imported but unused
./tests/app/main/views/test_send.py:18:1: F401 'tests.conftest.no_letter_contact_blocks' imported but unused
./tests/app/main/views/test_send.py:102:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_send.py:870:5: F841 local variable 'response' is assigned to but never used
./tests/app/main/views/test_send.py:1367:5: F841 local variable 'service_id' is assigned to but never used
./tests/app/main/views/test_send.py:1451:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:1620:80: E226 missing whitespace around arithmetic operator
./tests/app/main/views/test_send.py:1909:13: E126 continuation line over-indented for hanging indent
./tests/app/main/views/test_send.py:1912:9: E121 continuation line under-indented for hanging indent
./tests/app/main/views/test_service_settings.py:13:1: F811 redefinition of unused 'no_reply_to_email_addresses' from line 13
./tests/app/main/views/test_service_settings.py:13:1: F401 'tests.conftest.single_reply_to_email_address' imported but unused
./tests/app/main/views/test_service_settings.py:28:5: E123 closing bracket does not match indentation of opening bracket's line
./tests/app/main/views/test_service_settings.py:104:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:166:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:186:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:217:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:237:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:257:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:307:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:340:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:466:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:555:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:615:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:719:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:874:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:902:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:954:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:986:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1101:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1121:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1271:1: F811 redefinition of unused 'test_set_letter_contact_block_saves' from line 1189
./tests/app/main/views/test_service_settings.py:1433:5: F841 local variable 'page' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1495:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1540:5: F841 local variable 'mocked_get_fn' is assigned to but never used
./tests/app/main/views/test_service_settings.py:1570:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1589:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1621:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1641:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1658:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1676:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1697:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1759:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_service_settings.py:1775:1: F811 redefinition of unused 'single_reply_to_email_address' from line 13
./tests/app/main/views/test_templates.py:3:1: F401 'uuid' imported but unused
./tests/app/main/views/test_templates.py:11:1: F401 'tests.conftest.mock_get_user' imported but unused
./tests/app/main/views/test_templates.py:514:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:672:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:795:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_templates.py:835:1: F811 redefinition of unused 'mock_get_user' from line 11
./tests/app/main/views/test_two_factor.py:67:13: E126 continuation line over-indented for hanging indent
./tests/app/notify_client/test_notification_client.py:79:5: F841 local variable 'mock_post' is assigned to but never used
```
1. https://gds-way.cloudapps.digital/manuals/programming-languages/python/linting.html#how-to-use-flake8
2. d5ab8afef4/.flake8
In user research, we’ve seen users copy/pasting the contents of the
inbound SMS page into a spreadsheet, in order to keep a record of the
messages they receive. They even went as far as to write a macro which
fixed the errors caused by copying and pasting.
It would be much easier if we just gave them the data already in a
spreadsheet format. Which is what this commit does.
One caveat is that, because spreadsheets can contain executable code (ie
formulas), and because we’re populating the spreadsheet with
user-submitted data (albeit via SMS) we need to be careful about
injection attacks.
The details of how these attacks work are detailed here (interesting
reading): http://georgemauer.net/2017/10/07/csv-injection.html
The mitigation is to not allow characters which initialise a formula
at the start of the cell.
`test_utils.py` is actually a file for testing some utils, not
some utils for testing.
Test helper functions and fixtures are better placed in conftest,
and also means that we can use them in conftest.
Only services that have inbound SMS turned on should be able to see the
dashboard and ‘Received messages’ page.
There’s probably a cleaner way (decorator) of doing this permissions
stuff, but I think it can wait until we ship this.
