We already filter the usage-by-month query by financial year. When we
show the total usage for a service, we should be able to filter this
by financial year.
Then, when the two lots of data are put side by side, it all adds up.
This PR changes the response to POST /notifications/sms/<mmg | firetext> from a 400 response to a 200 response.
If we get a callback for a notification more than once or for a notification we log that but we return a 200 success response to the provider.
We have found that there is a situation where the send to provider throws a timeout exception but the provider did get the message, but we still send it to them again.
In which case they send the message twice, and callback for the message twice.
Another case where we may get duplicate callbacks is that the network gave the provider two callbacks meaning they pass those two callbacks onto us.
So it is really difficult to know if we sent to the provider twice or just got two callbacks.
The test_callback has many changes because I took the opportunity to use the client conftest fixture rather than the notify_api fixture.
The only 2 tests really changed are test_mmg_callback_returns_200_when_notification_id_not_found_or_already_updated and test_firetext_callback_returns_200_when_notification_id_not_found_or_already_updated
Previously research mode created a task to fake the callback from the providers. This meant there is an extra task for each notification than would be generated in a live situation.
This PR changes that, so that a research mode/test key notification calls the callback API rather than make a task to do that .
This ensures that the flow for research mode more closely mimics that of live, and removes a task from the process so we can more accurately test throughput,
the `to` field stores either the phone number or the email address
of the recipient - it's a bit more complicated for letters, since
there are address lines 1 through 6, and a postcode. In utils, they're
stored alongside the personalisation, and we have to ensure that when
we persist to the database we keep as much parity with utils to make
our work easier. Aside from sending, the `to` field is also used to
show recipients on the front end report pages - we've decided that the
best thing to store here is address_line_1 - which is probably going to
be either a person's name, company name, or PO box number
Also, a lot of tests and test cleanup - I added create_template and
create_notification functions in db.py, so if you're creating new
fixtures you can use these functions, and you won't need to pass
notify_db and notify_db_session around, huzzah!
also removed create param from sample_notification since it's not used
anywhere
it's almost entirely duplicated so share it across.
also clean up retrying - `task.retry(...)` raises a
celery.exceptions.Retry object, so you do not need to `raise` its
response. additionally, cleaned up tests around that since raising
Exception and asserting Exception is raised is dangerous as it could
mask actual programming errors
This meant that the research mode task would never work.
Also the way we mark data as "temporary-failure" with firetext is with first a pending status callback then a declined callback.
This PR changes the research mode task to account for that situation.
Right now we strip HTML from templates at the point of saving them. This
also converts stuff like ampersands to their entity form (eg &) and
this is what we save in the database.
This is a bad idea when you’re sending a text message or a letter, in
which an HTML entity makes no sense. But we still need to encode HTML in
the body of HTML emails.
The right place to do this is when rendering the templates. The code to
do this is now in utils. So this commit:
- pull in this new utils code
- removes the old
- adds some integration tests to make sure that everything is working
as expected (more thorough unit tests are happening in utils)
We are using the notify queue in this iteration because that queue is a low volume queue with it's own dedicated workers. This just saves us from building a new queue at this point, and a new queue may not be necessary.
- Added the `simulate` notification logic to version 2. We have 3 email addresses and phone numbers that are used
to simulate a successful post to /notifications. This was missed out of the version 2 endpoint.
- Added a test to template_dao to check for the default value of normal for new templates
- in v2 get_notifications, casted the path param to a uuid, if not uuid abort(404)
We’ve seen quite a few developers encounter the `Invalid token: expired`
error message when they’re getting started using the Notify API. When
this happens they either raise a support ticket or ask for help on
Slack.
In every case this has been because the clock on their
machine/environment/container isn’t accurate. The error message doesn’t
help them figure this out.
This commit adds extra detail to the error message so they can fix the
problem without having to come to us for help.
If the template is marked as priority the notification will be sent using the `notify` queue.
The `notify` queue is a low volume queue, messages here will not be queue behind a large job and should be delivered with in a more consistent time frame.
- Added templates.process_type and templates_history.process_type column.
- Added a template_process_type table to handle the enum for templates.process_type, initial values are normal and priority
https://www.pivotaltracker.com/story/show/135429147
while it's nice to use the decorator to signify fixtures with side
effects, it has unfortunate problems of completely overriding any
fixtures you've declared in the funcargs - so isn't really suitable
for our usecase where we often have other fixtures we rely on to
return values to us.
So for consistency, let's remove this and stick to using funcargs
to define our fixtures
when you invoke the fixture `sample_user`, it does two things: it
creates the user in the database, but also returns the user, a useful
object that you may want to manipulate or reference in your test.
however, when you invoke the fixture `notify_db_session`, it doesn't
do anything - rather, it *promises* to clear up the database tables
at the end of the test run.
because we have no need of the notify_db_session object in our tests
(indeed, for a long time this fixture just returned `None`), using
`pytest.mark.usefixtures('notify_db_session')` brings attention to the
fact that this is a side-effect fixture rather than a data setup
fixture. Functionally it is identical to passing as a parameter
note that all of these tests have to be checked to ensure that they
still call through to notify_db_session (notify_db not required) to
tear down the database after the test runs - since it's no longer
required to pass it in to the function just to invoke the sample_user
function
this create_user function can be imported for use in creating specific
users in your tests, for example
```
from tests.app.db import create_user
def test_create_user_persists_to_database(notify_db_session):
user = create_user()
assert User.query.count() == 1
```
this has the benefit of not requiring you to pass the notify_db and
notify_db_session fixtures around, and separating custom object
creation from the fixture dependency trees to aid clarity
I started with sample_user since it has no downstream dependencies, but
the intention is to push this out to all db fixtures eventually. This
is a total conversion, but can be rolled out in a non-breaking manner
by keeping arguments in the fixture, and passing them through to the
new db function - then tests can be updated to use the create_* instead
of sample_* functions as and when you want to
brings in a fix to InvalidEmail/Phone/AddressExceptions not being
instantiated correctly. `exception.message` is not a python standard,
so we shouldn't be relying on it to transmit exception reasons -
rather we should be using `str(exception)` instead. This involved a
handful of small changes to the schema validation
