make 500s change priorities quicker

it's not acceptable for a constantly failing provider to take 50 minutes
to drain (5x reducing priority by 10). But similarly, we need _some_
delay, or a handful of concurrent failures will completely turn off a
provider, rendering the whole excercise kinda pointless. Setting the
delay before it tries to reduce priority again to one minute is nice
because it means that if one request times out and returns 502, then any
other requests that are in flight at that time will time out before the
one minute is up and not switch, but any requests made after the switch
that take sixty seconds to time out will affect it.
This commit is contained in:
Leo Hemsted
2019-11-20 17:23:39 +00:00
parent 2d7bf664f5
commit f7fbd6de5b
6 changed files with 28 additions and 19 deletions

View File

@@ -1,6 +1,6 @@
import random
from urllib import parse
from datetime import datetime
from datetime import datetime, timedelta
from flask import current_app
from notifications_utils.recipients import (
@@ -66,7 +66,7 @@ def send_sms_to_provider(notification):
except Exception as e:
notification.billable_units = template.fragment_count
dao_update_notification(notification)
dao_reduce_sms_provider_priority(provider.get_name())
dao_reduce_sms_provider_priority(provider.get_name(), time_threshold=timedelta(minutes=1))
raise e
else:
notification.billable_units = template.fragment_count