add raw request timings to provider send functions

we're using statsd to monitor how long provider requests are taking.
However, there's lots of busy work that happens inside our statsd
metrics timing window. Things like json dumping and loading, building
headers, exception handling, etc.

for firetext/mmg, the response object from requests has an elapsed
property [1], which captures from sending raw data to parsing the
response headers. for ses, it's a bit trickier, but boto3 exposes a few
event hooks [2]. it's hard to find them without stepping through the
code, but the interesting ones are before-call, after-call,
after-call-error, request-created, and response-received. The
before-call and after-call involve some marshalling, built-in retrying,
etc, while request-created and response-received are much lower level.
They might be called more than once per ses request, if boto3 itself
retries the request on 5xx, 429 and low level socket errors [3].

Add these as new `raw-request-time` metrics rather than overwriting to
avoid changing the meaning of an existing metric, and to let us compare
the metrics to see if there's a noticeable difference at all

[1] https://requests.readthedocs.io/en/master/api/#requests.Response.elapsed
[2] https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
[3] https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#legacy-retry-mode
This commit is contained in:
Leo Hemsted
2020-05-29 13:56:29 +01:00
parent 5462087f21
commit f2f2509c9b
3 changed files with 27 additions and 0 deletions

View File

@@ -97,6 +97,7 @@ class FiretextClient(SmsClient):
"reference": reference
}
response = None
start_time = monotonic()
try:
response = request(
@@ -121,4 +122,6 @@ class FiretextClient(SmsClient):
elapsed_time = monotonic() - start_time
self.current_app.logger.info("Firetext request for {} finished in {}".format(reference, elapsed_time))
self.statsd_client.timing("clients.firetext.request-time", elapsed_time)
if response and hasattr(response, 'elapsed'):
self.statsd_client.timing("clients.firetext.raw-request-time", response.elapsed.total_seconds())
return response

View File

@@ -71,6 +71,7 @@ class MMGClient(SmsClient):
"multi": multi
}
response = None
start_time = monotonic()
try:
response = request(
@@ -97,6 +98,9 @@ class MMGClient(SmsClient):
finally:
elapsed_time = monotonic() - start_time
self.statsd_client.timing("clients.mmg.request-time", elapsed_time)
if response and hasattr(response, 'elapsed'):
self.statsd_client.timing("clients.mmg.raw-request-time", response.elapsed.total_seconds())
self.current_app.logger.info("MMG request for {} finished in {}".format(reference, elapsed_time))
return response