darkhelm/notifications-api
1
0
Fork 0
mirror of https://github.com/GSA/notifications-api.git synced 2025-12-20 07:21:13 -05:00
Code Issues Packages Projects Releases Wiki Activity

Close DB connection whilst making HTTP to SMS providers

Browse Source 
At the moment, when we are processing and sending an SMS we open
a DB connection at the start of the celery task and then close it
at the end of the celery task. Nice and simple.

However, during that celery task we make an HTTP call out to our
SMS providers. If our SMS providers have problems or response times
start to slow then it means we have an open DB connection sat waiting
for our SMS providers to respond which could take seconds. If our
SMS providers grind to a halt, this would cause all of the
celery tasks to hold on to their connections and we would run out
of DB connections and Notify would fall over.

We think we can solve this by closing the DB session which releases
the DB connection back to the pool.

Note, we've seen this happen in staging during load testing if our
SMS provider stub has fallen over. We've never seen it in production
and it may be less unlikely to happen as we are balancing traffic
across two providers and they generally have very good uptime.

One downside to be aware of is there could be a slight increase in
time spent to send an SMS as we will now spend a bit of extra time
closing the DB session and then reopening it again after the HTTP
request is done.

Note, there is no reason this approach couldn't be copied for our
email provider too if it appears successful.
This commit is contained in:
David McDonald
2021-12-21 15:30:28 +00:00
parent 7dd3e1fa87
commit 2584946823
2 changed files with 61 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
app/delivery/send_to_providers.py
View File

@@ -10,7 +10,7 @@ from notifications_utils.template import (
SMSMessageTemplate,
)
from app import create_uuid, notification_provider_clients, statsd_client
from app import create_uuid, db, notification_provider_clients, statsd_client
from app.celery.research_mode_tasks import (
send_email_response,
send_sms_response,
@@ -64,13 +64,21 @@ def send_sms_to_provider(notification):
else:
try:
provider.send_sms(
to=notification.normalised_to,
content=str(template),
reference=str(notification.id),
sender=notification.reply_to_text,
international=notification.international
)
# End DB session here so that we don't have a connection stuck open waiting on the call
# to one of the SMS providers
# We don't want to tie our DB connections being open to the performance of our SMS
# providers as a slow down of our providers can cause us to run out of DB connections
# Therefore we pull all the data from our DB models into `send_sms_kwargs`now before
# closing the session (as otherwise it would be reopened immediately)
send_sms_kwargs = {
'to': notification.normalised_to,
'content': str(template),
'reference': str(notification.id),
'sender': notification.reply_to_text,
'international': notification.international,
}
db.session.close() # no commit needed as no changes to objects have been made above
provider.send_sms(**send_sms_kwargs)
except Exception as e:
notification.billable_units = template.fragment_count
dao_update_notification(notification)