darkhelm/notifications-admin
1
0
Fork 0
mirror of https://github.com/GSA/notifications-admin.git synced 2026-02-21 02:44:42 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
203393de235631db0ce3c38dec85da170a634fba
notifications-admin/tests/app/notify_client/test_service_api_client.py

433 lines
14 KiB
Python
Raw Normal View History

Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
from unittest.mock import call
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
from uuid import uuid4
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
pass 'today_only' flag to the back-end from send page also bump notification-utils requirement to 8.7.0 for changes to RecipientCSV
2016-07-25 15:26:43 +01:00
import pytest
Delete caches when user accepts invite Accepting an invite changes: - the `user_to_service` list of users returned by `GET /service/<id>` - the `services` list return by `GET /user/<id>` The latter change is causing the functional tests to fail.
2018-04-19 13:15:52 +01:00
from app import invite_api_client, service_api_client, user_api_client
rename api_client to service_api_client
2016-07-15 15:23:23 +01:00
from app.notify_client.service_api_client import ServiceAPIClient
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
from tests.conftest import SERVICE_ONE_ID
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
FAKE_TEMPLATE_ID = uuid4()
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
def test_client_posts_archived_true_when_deleting_template(mocker):
Make `_attach_current_user` a pure function Mutating dictionaries is gross and doesn’t work as you’d expect. Better to have the function return a new dictionary instead. Means we can be explicit that `created_by` is one of the allowed params when updating a service.
2016-08-11 14:20:43 +01:00
mocker.patch('app.notify_client.current_user', id='1')
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
expected_data = {
'archived': True,
Make `_attach_current_user` a pure function Mutating dictionaries is gross and doesn’t work as you’d expect. Better to have the function return a new dictionary instead. Means we can be explicit that `created_by` is one of the allowed params when updating a service.
2016-08-11 14:20:43 +01:00
'created_by': '1'
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
}
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
expected_url = '/service/{}/template/{}'.format(SERVICE_ONE_ID, FAKE_TEMPLATE_ID)
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
client = ServiceAPIClient()
rename api_client to service_api_client
2016-07-15 15:23:23 +01:00
mock_post = mocker.patch('app.notify_client.service_api_client.ServiceAPIClient.post')
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
client.delete_service_template(SERVICE_ONE_ID, FAKE_TEMPLATE_ID)
delete template button POSTs archived:true
2016-05-23 13:59:33 +01:00
mock_post.assert_called_once_with(expected_url, data=expected_data)
add tests for service_api_client
2016-07-21 17:32:28 +01:00
use service statistics endpoint instead of detailed service we're not actually looking at the detailed service aspects - just the stats. We're doing this in three places: * dashboard * notification activity page * when checking jobs to see if we're over the daily limit change these places to use a new api endpoint (service/id/statistics), which hopefully be a little more performant, and will definitely be a little more organised - moving away from generic endpoints with loads of optional parameters. We still need the detailed endpoints for the platform admin page tho. Depends on https://github.com/alphagov/notifications-api/pull/1865
2018-05-09 13:53:02 +01:00
def test_client_gets_service(mocker):
add tests for service_api_client
2016-07-21 17:32:28 +01:00
client = ServiceAPIClient()
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
mock_get = mocker.patch.object(client, 'get', return_value={})
add tests for service_api_client
2016-07-21 17:32:28 +01:00
use service statistics endpoint instead of detailed service we're not actually looking at the detailed service aspects - just the stats. We're doing this in three places: * dashboard * notification activity page * when checking jobs to see if we're over the daily limit change these places to use a new api endpoint (service/id/statistics), which hopefully be a little more performant, and will definitely be a little more organised - moving away from generic endpoints with loads of optional parameters. We still need the detailed endpoints for the platform admin page tho. Depends on https://github.com/alphagov/notifications-api/pull/1865
2018-05-09 13:53:02 +01:00
client.get_service('foo')
mock_get.assert_called_once_with('/service/foo')
Add limit_days argument to statistics API client method Allows getting notification counts for a given number of days to support services with custom data retention periods (admin dashboard page should still display counts for the last 7 days, while the notifications page displays all stored notifications).
2018-08-13 17:04:40 +01:00
@pytest.mark.parametrize('today_only, limit_days', [
(True, None),
(False, None),
(False, 30),
])
def test_client_gets_service_statistics(mocker, today_only, limit_days):
use service statistics endpoint instead of detailed service we're not actually looking at the detailed service aspects - just the stats. We're doing this in three places: * dashboard * notification activity page * when checking jobs to see if we're over the daily limit change these places to use a new api endpoint (service/id/statistics), which hopefully be a little more performant, and will definitely be a little more organised - moving away from generic endpoints with loads of optional parameters. We still need the detailed endpoints for the platform admin page tho. Depends on https://github.com/alphagov/notifications-api/pull/1865
2018-05-09 13:53:02 +01:00
client = ServiceAPIClient()
mock_get = mocker.patch.object(client, 'get', return_value={'data': {'a': 'b'}})
Add limit_days argument to statistics API client method Allows getting notification counts for a given number of days to support services with custom data retention periods (admin dashboard page should still display counts for the last 7 days, while the notifications page displays all stored notifications).
2018-08-13 17:04:40 +01:00
ret = client.get_service_statistics('foo', today_only, limit_days)
use service statistics endpoint instead of detailed service we're not actually looking at the detailed service aspects - just the stats. We're doing this in three places: * dashboard * notification activity page * when checking jobs to see if we're over the daily limit change these places to use a new api endpoint (service/id/statistics), which hopefully be a little more performant, and will definitely be a little more organised - moving away from generic endpoints with loads of optional parameters. We still need the detailed endpoints for the platform admin page tho. Depends on https://github.com/alphagov/notifications-api/pull/1865
2018-05-09 13:53:02 +01:00
assert ret == {'a': 'b'}
Add limit_days argument to statistics API client method Allows getting notification counts for a given number of days to support services with custom data retention periods (admin dashboard page should still display counts for the last 7 days, while the notifications page displays all stored notifications).
2018-08-13 17:04:40 +01:00
mock_get.assert_called_once_with('/service/foo/statistics', params={
'today_only': today_only, 'limit_days': limit_days
})
Only allow update service to modify named attrs To prevent typos and inadvertently updating something we shouldn’t, this adds some filtering to the update_service method to make sure it is only allowed to update certain attributes of a service.
2016-08-11 13:55:42 +01:00
def test_client_only_updates_allowed_attributes(mocker):
Make `_attach_current_user` a pure function Mutating dictionaries is gross and doesn’t work as you’d expect. Better to have the function return a new dictionary instead. Means we can be explicit that `created_by` is one of the allowed params when updating a service.
2016-08-11 14:20:43 +01:00
mocker.patch('app.notify_client.current_user', id='1')
Only allow update service to modify named attrs To prevent typos and inadvertently updating something we shouldn’t, this adds some filtering to the update_service method to make sure it is only allowed to update certain attributes of a service.
2016-08-11 13:55:42 +01:00
with pytest.raises(TypeError) as error:
ServiceAPIClient().update_service('service_id', foo='bar')
assert str(error.value) == 'Not allowed to update service attributes: foo'
Collect organisation type So that we can default services to their appropriate text allowance, we need to find out what sector they're in. So let's start collecting that from teams as they create new services. I think Central/Local/NHS are the right options, but these can be easily changed if not.
2017-10-04 11:49:32 +01:00
def test_client_creates_service_with_correct_data(
mocker,
active_user_with_permissions,
fake_uuid,
):
client = ServiceAPIClient()
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
mock_post = mocker.patch.object(client, 'post', return_value={'data': {'id': None}})
Collect organisation type So that we can default services to their appropriate text allowance, we need to find out what sector they're in. So let's start collecting that from teams as they create new services. I think Central/Local/NHS are the right options, but these can be easily changed if not.
2017-10-04 11:49:32 +01:00
mocker.patch('app.notify_client.current_user', id='123')
client.create_service(
Clear user cache when deleting a service The user JSON has a list of service IDs
2018-04-19 13:23:47 +01:00
'My first service',
'central_government',
1,
True,
fake_uuid,
'test@example.com',
Pass service domain to api when adding a new service We need to pass the domain to api when adding a service so that api can link the domain of the service with a letter brand.
2019-02-12 14:02:21 +00:00
'nhs.uk'
Collect organisation type So that we can default services to their appropriate text allowance, we need to find out what sector they're in. So let's start collecting that from teams as they create new services. I think Central/Local/NHS are the right options, but these can be easily changed if not.
2017-10-04 11:49:32 +01:00
)
mock_post.assert_called_once_with(
'/service',
dict(
# Autogenerated arguments
created_by='123',
active=True,
# ‘service_name’ argument is coerced to ‘name’
name='My first service',
# The rest pass through with the same names
organisation_type='central_government',
message_limit=1,
restricted=True,
user_id=fake_uuid,
email_from='test@example.com',
Pass service domain to api when adding a new service We need to pass the domain to api when adding a service so that api can link the domain of the service with a letter brand.
2019-02-12 14:02:21 +00:00
service_domain='nhs.uk'
Collect organisation type So that we can default services to their appropriate text allowance, we need to find out what sector they're in. So let's start collecting that from teams as they create new services. I think Central/Local/NHS are the right options, but these can be easily changed if not.
2017-10-04 11:49:32 +01:00
),
)
Add API client method to count templates When users request to go live we check stuff like: - if they’ve added templates - if they have email templates (then we can check their reply to address) This commit adds a method to do this programatically rather than manually. We _could_ do this in SQL, but for page that’s used intermittently it doesn’t feel worth the work/optimisation (and the client method is at least in place now if we do ever need to lean on this code more heavily).
2018-02-21 14:37:27 +00:00
@pytest.mark.parametrize('template_data, extra_args, expected_count', (
(
[],
{},
0,
),
(
[],
{'template_type': 'email'},
0,
),
(
[
{'template_type': 'email'},
{'template_type': 'sms'},
],
{},
2,
),
(
[
{'template_type': 'email'},
{'template_type': 'sms'},
],
{'template_type': 'email'},
1,
),
(
[
{'template_type': 'email'},
{'template_type': 'sms'},
],
{'template_type': 'letter'},
0,
),
))
def test_client_returns_count_of_service_templates(
app_,
mocker,
template_data,
extra_args,
expected_count,
):
mocker.patch(
'app.service_api_client.get_service_templates',
return_value={'data': template_data}
)
assert service_api_client.count_service_templates(
SERVICE_ONE_ID, **extra_args
) == expected_count
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
@pytest.mark.parametrize(
(
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
'client_method,'
'extra_args,'
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
'expected_cache_get_calls,'
'cache_value,'
'expected_api_calls,'
'expected_cache_set_calls,'
'expected_return_value,'
),
[
(
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
service_api_client.get_service,
[SERVICE_ONE_ID],
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
[
call('service-{}'.format(SERVICE_ONE_ID))
],
b'{"data_from": "cache"}',
[],
[],
{'data_from': 'cache'},
),
(
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
service_api_client.get_service,
[SERVICE_ONE_ID],
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
[
call('service-{}'.format(SERVICE_ONE_ID))
],
None,
[
use service statistics endpoint instead of detailed service we're not actually looking at the detailed service aspects - just the stats. We're doing this in three places: * dashboard * notification activity page * when checking jobs to see if we're over the daily limit change these places to use a new api endpoint (service/id/statistics), which hopefully be a little more performant, and will definitely be a little more organised - moving away from generic endpoints with loads of optional parameters. We still need the detailed endpoints for the platform admin page tho. Depends on https://github.com/alphagov/notifications-api/pull/1865
2018-05-09 13:53:02 +01:00
call('/service/{}'.format(SERVICE_ONE_ID))
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
],
[
call(
'service-{}'.format(SERVICE_ONE_ID),
'{"data_from": "api"}',
Make Redis hold onto cached API responses longer Redis is giving us a big performance boost (it’s roughly halved the median request time on the admin app). Once we’re confident that it’s working properly[1] we can eke out a bit more performance from it by keeping the caches alive for longer. As far as I can tell we’re still using Redis in a very low-volume way[2], so increasing the number of things we’re storing shouldn’t start taxing our Redis server at all. But reducing the number of times we have to hit the API to refresh the cache _should_ result in some performance increase. --- 1. ie we’re not seeing instances of stale caches not being invalidated 2. We have 2.5G of available space in Redis. Here is our current usage: ``` used_memory:7728960 used_memory_human:7.37M used_memory_rss:7728960 used_memory_peak:16563776 used_memory_peak_human:15.79M used_memory_lua:37888 ```
2018-04-23 17:07:41 +01:00
ex=604800,
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
)
],
{'data_from': 'api'},
),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
(
service_api_client.get_service_template,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID],
[
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
call('template-{}-version-None'.format(FAKE_TEMPLATE_ID))
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
],
b'{"data_from": "cache"}',
[],
[],
{'data_from': 'cache'},
),
(
service_api_client.get_service_template,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID],
[
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
call('template-{}-version-None'.format(FAKE_TEMPLATE_ID))
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
],
None,
[
call('/service/{}/template/{}'.format(SERVICE_ONE_ID, FAKE_TEMPLATE_ID))
],
[
call(
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
'{"data_from": "api"}',
Make Redis hold onto cached API responses longer Redis is giving us a big performance boost (it’s roughly halved the median request time on the admin app). Once we’re confident that it’s working properly[1] we can eke out a bit more performance from it by keeping the caches alive for longer. As far as I can tell we’re still using Redis in a very low-volume way[2], so increasing the number of things we’re storing shouldn’t start taxing our Redis server at all. But reducing the number of times we have to hit the API to refresh the cache _should_ result in some performance increase. --- 1. ie we’re not seeing instances of stale caches not being invalidated 2. We have 2.5G of available space in Redis. Here is our current usage: ``` used_memory:7728960 used_memory_human:7.37M used_memory_rss:7728960 used_memory_peak:16563776 used_memory_peak_human:15.79M used_memory_lua:37888 ```
2018-04-23 17:07:41 +01:00
ex=604800,
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
)
],
{'data_from': 'api'},
),
(
service_api_client.get_service_template,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID, 1],
[
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
call('template-{}-version-1'.format(FAKE_TEMPLATE_ID))
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
],
b'{"data_from": "cache"}',
[],
[],
{'data_from': 'cache'},
),
(
service_api_client.get_service_template,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID, 1],
[
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
call('template-{}-version-1'.format(FAKE_TEMPLATE_ID))
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
],
None,
[
call('/service/{}/template/{}/version/1'.format(SERVICE_ONE_ID, FAKE_TEMPLATE_ID))
],
[
call(
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'template-{}-version-1'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
'{"data_from": "api"}',
Make Redis hold onto cached API responses longer Redis is giving us a big performance boost (it’s roughly halved the median request time on the admin app). Once we’re confident that it’s working properly[1] we can eke out a bit more performance from it by keeping the caches alive for longer. As far as I can tell we’re still using Redis in a very low-volume way[2], so increasing the number of things we’re storing shouldn’t start taxing our Redis server at all. But reducing the number of times we have to hit the API to refresh the cache _should_ result in some performance increase. --- 1. ie we’re not seeing instances of stale caches not being invalidated 2. We have 2.5G of available space in Redis. Here is our current usage: ``` used_memory:7728960 used_memory_human:7.37M used_memory_rss:7728960 used_memory_peak:16563776 used_memory_peak_human:15.79M used_memory_lua:37888 ```
2018-04-23 17:07:41 +01:00
ex=604800,
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
)
],
{'data_from': 'api'},
),
Add tests for all templates and template versions
2018-04-20 17:29:06 +01:00
(
service_api_client.get_service_templates,
[SERVICE_ONE_ID],
[
call('service-{}-templates'.format(SERVICE_ONE_ID))
],
b'{"data_from": "cache"}',
[],
[],
{'data_from': 'cache'},
),
(
service_api_client.get_service_templates,
[SERVICE_ONE_ID],
[
call('service-{}-templates'.format(SERVICE_ONE_ID))
],
None,
[
call('/service/{}/template'.format(SERVICE_ONE_ID))
],
[
call(
'service-{}-templates'.format(SERVICE_ONE_ID),
'{"data_from": "api"}',
Make Redis hold onto cached API responses longer Redis is giving us a big performance boost (it’s roughly halved the median request time on the admin app). Once we’re confident that it’s working properly[1] we can eke out a bit more performance from it by keeping the caches alive for longer. As far as I can tell we’re still using Redis in a very low-volume way[2], so increasing the number of things we’re storing shouldn’t start taxing our Redis server at all. But reducing the number of times we have to hit the API to refresh the cache _should_ result in some performance increase. --- 1. ie we’re not seeing instances of stale caches not being invalidated 2. We have 2.5G of available space in Redis. Here is our current usage: ``` used_memory:7728960 used_memory_human:7.37M used_memory_rss:7728960 used_memory_peak:16563776 used_memory_peak_human:15.79M used_memory_lua:37888 ```
2018-04-23 17:07:41 +01:00
ex=604800,
Add tests for all templates and template versions
2018-04-20 17:29:06 +01:00
)
],
{'data_from': 'api'},
),
(
service_api_client.get_service_template_versions,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID],
[
call('template-{}-versions'.format(FAKE_TEMPLATE_ID))
],
b'{"data_from": "cache"}',
[],
[],
{'data_from': 'cache'},
),
(
service_api_client.get_service_template_versions,
[SERVICE_ONE_ID, FAKE_TEMPLATE_ID],
[
call('template-{}-versions'.format(FAKE_TEMPLATE_ID))
],
None,
[
call('/service/{}/template/{}/versions'.format(SERVICE_ONE_ID, FAKE_TEMPLATE_ID))
],
[
call(
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
'{"data_from": "api"}',
Make Redis hold onto cached API responses longer Redis is giving us a big performance boost (it’s roughly halved the median request time on the admin app). Once we’re confident that it’s working properly[1] we can eke out a bit more performance from it by keeping the caches alive for longer. As far as I can tell we’re still using Redis in a very low-volume way[2], so increasing the number of things we’re storing shouldn’t start taxing our Redis server at all. But reducing the number of times we have to hit the API to refresh the cache _should_ result in some performance increase. --- 1. ie we’re not seeing instances of stale caches not being invalidated 2. We have 2.5G of available space in Redis. Here is our current usage: ``` used_memory:7728960 used_memory_human:7.37M used_memory_rss:7728960 used_memory_peak:16563776 used_memory_peak_human:15.79M used_memory_lua:37888 ```
2018-04-23 17:07:41 +01:00
ex=604800,
Add tests for all templates and template versions
2018-04-20 17:29:06 +01:00
)
],
{'data_from': 'api'},
),
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
]
)
def test_returns_value_from_cache(
mocker,
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
client_method,
extra_args,
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
expected_cache_get_calls,
cache_value,
expected_return_value,
expected_api_calls,
expected_cache_set_calls,
):
mock_redis_get = mocker.patch(
'app.notify_client.RedisClient.get',
return_value=cache_value,
)
mock_api_get = mocker.patch(
'app.notify_client.NotifyAdminAPIClient.get',
return_value={'data_from': 'api'},
)
mock_redis_set = mocker.patch(
'app.notify_client.RedisClient.set',
)
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
assert client_method(*extra_args) == expected_return_value
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
assert mock_redis_get.call_args_list == expected_cache_get_calls
assert mock_api_get.call_args_list == expected_api_calls
assert mock_redis_set.call_args_list == expected_cache_set_calls
@pytest.mark.parametrize('client, method, extra_args, extra_kwargs', [
(service_api_client, 'update_service', [SERVICE_ONE_ID], {'name': 'foo'}),
(service_api_client, 'update_service_with_properties', [SERVICE_ONE_ID], {'properties': {}}),
(service_api_client, 'archive_service', [SERVICE_ONE_ID], {}),
(service_api_client, 'suspend_service', [SERVICE_ONE_ID], {}),
(service_api_client, 'resume_service', [SERVICE_ONE_ID], {}),
(service_api_client, 'remove_user_from_service', [SERVICE_ONE_ID, ''], {}),
(service_api_client, 'update_whitelist', [SERVICE_ONE_ID, {}], {}),
(service_api_client, 'create_service_inbound_api', [SERVICE_ONE_ID] + [''] * 3, {}),
(service_api_client, 'update_service_inbound_api', [SERVICE_ONE_ID] + [''] * 4, {}),
(service_api_client, 'add_reply_to_email_address', [SERVICE_ONE_ID, ''], {}),
(service_api_client, 'update_reply_to_email_address', [SERVICE_ONE_ID] + [''] * 2, {}),
Update service_api_client to use new endpoints API now has separate endpoints to archive email reply-to addresses and SMS senders, so we no longer need to use the endpoints for updating.
2018-04-26 15:09:37 +01:00
(service_api_client, 'delete_reply_to_email_address', [SERVICE_ONE_ID, ''], {}),
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
(service_api_client, 'add_letter_contact', [SERVICE_ONE_ID, ''], {}),
(service_api_client, 'update_letter_contact', [SERVICE_ONE_ID] + [''] * 2, {}),
(service_api_client, 'add_sms_sender', [SERVICE_ONE_ID, ''], {}),
(service_api_client, 'update_sms_sender', [SERVICE_ONE_ID] + [''] * 2, {}),
Update service_api_client to use new endpoints API now has separate endpoints to archive email reply-to addresses and SMS senders, so we no longer need to use the endpoints for updating.
2018-04-26 15:09:37 +01:00
(service_api_client, 'delete_sms_sender', [SERVICE_ONE_ID, ''], {}),
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
(service_api_client, 'update_service_callback_api', [SERVICE_ONE_ID] + [''] * 4, {}),
(service_api_client, 'create_service_callback_api', [SERVICE_ONE_ID] + [''] * 3, {}),
Stop calling `fake_uuid` fixture directly Pytest is deprecating the direct calling of fixtures. One fixture that we call directly quite a lot is `fake_uuid`. Since it just returns the value of `sample_uuid()` we can either call that instead (where we need a fixed value) or generate a new UUID each time (where a fixed value is not needed).
2018-09-26 16:41:04 +01:00
(user_api_client, 'add_user_to_service', [SERVICE_ONE_ID, uuid4(), []], {}),
(invite_api_client, 'accept_invite', [SERVICE_ONE_ID, uuid4()], {}),
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
])
def test_deletes_service_cache(
app_,
mock_get_user,
mocker,
client,
method,
extra_args,
extra_kwargs,
):
mocker.patch('app.notify_client.current_user', id='1')
mock_redis_delete = mocker.patch('app.notify_client.RedisClient.delete')
mock_request = mocker.patch('notifications_python_client.base.BaseAPIClient.request')
getattr(client, method)(*extra_args, **extra_kwargs)
Cache `GET /user` response in Redis In the same way, and for the same reasons that we’re caching the service object. Here’s a sample of the data returned by the API – so we should make sure that any changes to this data invalidate the cache. If we ever change a user’s phone number (for example) directly in the database, then we will need to invalidate this cache manually. ```python {      'data':{         'organisations':[            '4c707b81-4c6d-4d33-9376-17f0de6e0405'       ],       'logged_in_at':'2018-04-10T11:41:03.781990Z',       'id':'2c45486e-177e-40b8-997d-5f4f81a461ca',       'email_address':'test@example.gov.uk',       'platform_admin':False,       'password_changed_at':'2018-01-01 10:10:10.100000',       'permissions':{            '42a9d4f2-1444-4e22-9133-52d9e406213f':[               'manage_api_keys',             'send_letters',             'manage_users',             'manage_templates',             'view_activity',             'send_texts',             'send_emails',             'manage_settings'          ],          'a928eef8-0f25-41ca-b480-0447f29b2c20':[               'manage_users',             'manage_templates',             'manage_settings',             'send_texts',             'send_emails',             'send_letters',             'manage_api_keys',             'view_activity'          ],       },       'state':'active',       'mobile_number':'07700900123',       'failed_login_count':0,       'name':'Example',       'services':[            '6078a8c0-52f5-4c4f-b724-d7d1ff2d3884',          '6afe3c1c-7fda-4d8d-aa8d-769c4bdf7803',       ],       'current_session_id':'fea2ade1-db0a-4c90-93e7-c64a877ce83e',       'auth_type':'sms_auth'    } } ```
2018-04-10 13:30:52 +01:00
assert call('service-{}'.format(SERVICE_ONE_ID)) in mock_redis_delete.call_args_list
Add Redis cache between admin and API Most of the time spent by the admin app to generate a page is spent waiting for the API. This is slow for three reasons: 1. Talking to the API means going out to the internet, then through nginx, the Flask app, SQLAlchemy, down to the database, and then serialising the result to JSON and making it into a HTTP response 2. Each call to the API is synchronous, therefore if a page needs 3 API calls to render then the second API call won’t be made until the first has finished, and the third won’t start until the second has finished 3. Every request for a service page in the admin app makes a minimum of two requests to the API (`GET /service/…` and `GET /user/…`) Hitting the database will always be the slowest part of an app like Notify. But this slowness is exacerbated by 2. and 3. Conversely every speedup made to 1. is multiplied by 2. and 3. So this pull request aims to make 1. a _lot_ faster by taking nginx, Flask, SQLAlchemy and the database out of the equation. It replaces them with Redis, which as an in-memory key/value store is a lot faster than Postgres. There is still the overhead of going across the network to talk to Redis, but the net improvement is vast. This commit only caches the `GET /service` response, but is written in such a way that we can easily expand to caching other responses down the line. The tradeoff here is that our code is more complex, and we risk introducing edge cases where a cache becomes stale. The mitigations against this are: - invalidating all caches after 24h so a stale cache doesn’t remain around indefinitely - being careful when we add new stuff to the service response --- Some indicative numbers, based on: - `GET http://localhost:6012/services/<service_id>/template/<template_id>` - with the admin app running locally - talking to Redis running locally - also talking to the API running locally, itself talking to a local Postgres instance - times measured with Chrome web inspector, average of 10 requests ╲ | No cache | Cache service | Cache service and user | Cache service, user and template -- | -- | -- | -- | -- **Request time** | 136ms | 97ms | 73ms | 37ms **Improvement** | 0% | 41% | 88% | 265% --- Estimates of how much storage this requires: - Services: 1,942 on production × 2kb = 4Mb - Users: 4,534 on production × 2kb = 9Mb - Templates: 7,079 on production × 4kb = 28Mb
2018-04-06 13:37:49 +01:00
assert len(mock_request.call_args_list) == 1
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
@pytest.mark.parametrize('method, extra_args, expected_cache_deletes', [
('create_service_template', ['name', 'type_', 'content', SERVICE_ONE_ID], [
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'service-{}-templates'.format(SERVICE_ONE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
]),
('update_service_template', [FAKE_TEMPLATE_ID, 'foo', 'sms', 'bar', SERVICE_ONE_ID], [
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'service-{}-templates'.format(SERVICE_ONE_ID),
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
]),
('redact_service_template', [SERVICE_ONE_ID, FAKE_TEMPLATE_ID], [
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'service-{}-templates'.format(SERVICE_ONE_ID),
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
]),
('update_service_template_sender', [SERVICE_ONE_ID, FAKE_TEMPLATE_ID, 'foo'], [
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'service-{}-templates'.format(SERVICE_ONE_ID),
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
]),
Edit template postage from a separate view
2019-01-29 15:43:59 +00:00
('update_service_template_postage', [SERVICE_ONE_ID, FAKE_TEMPLATE_ID, 'first'], [
'service-{}-templates'.format(SERVICE_ONE_ID),
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
]),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
('delete_service_template', [SERVICE_ONE_ID, FAKE_TEMPLATE_ID], [
Rewrite cache decorator to use format string This is easier to read than having to understand the arguments 1…n of the cache decorator are ‘magic’, and gives us more flexibility about how the cache keys are formatted, eg being able to add words in the middle of them. Also changes the key format for all templates to be `service-{service_id}-templates` instead of `templates-{service_id}` because then it’s clearer what the ID represents.
2018-04-20 16:32:02 +01:00
'service-{}-templates'.format(SERVICE_ONE_ID),
'template-{}-version-None'.format(FAKE_TEMPLATE_ID),
'template-{}-versions'.format(FAKE_TEMPLATE_ID),
Add caching of templates in Redis A lot of the frequently-used pages in the admin app rely on the API to get templates. So this commit adds three new caches: - a single template version (including a key without a version number, which is the current version) - all the templates for a service - all versions of a template The first will be the most crucial for performance, but there’s not much cost to adding the other two.
2018-04-19 10:24:35 +01:00
]),
])
def test_deletes_caches_when_modifying_templates(
app_,
mock_get_user,
mocker,
method,
extra_args,
expected_cache_deletes,
):
mocker.patch('app.notify_client.current_user', id='1')
mock_redis_delete = mocker.patch('app.notify_client.RedisClient.delete')
mock_request = mocker.patch('notifications_python_client.base.BaseAPIClient.request')
getattr(service_api_client, method)(*extra_args)
assert mock_redis_delete.call_args_list == list(map(call, expected_cache_deletes))
assert len(mock_request.call_args_list) == 1
Reference in New Issue Copy Permalink