Commit Graph

8865 Commits

Author SHA1 Message Date
Ben Thorner
fe6afd18d6 Refactor tests for monthly usage API
These are now consistent with the yearly usage API tests.
2022-04-26 13:11:32 +01:00
Ben Thorner
4d3c604faf Move edge case usage API tests down to DAO level
These tests weren't checking anything structural about the APIs
beyond what's covered by the other tests. They represent edge
cases that we can check at a lower level instead.

It was also unclear what these tests were actually testing, as
the term "all cases" is vague. Looking at the test data, there
are variations in rates, multipliers and billable units for SMS
and letters, which I've summarised as "variable rates".

Note: I've removed part of the test data - for the first class
letter rate - as it's not clearly adding anything.
2022-04-26 13:09:25 +01:00
Ben Thorner
0a8d35f909 Remove redundant test for monthly usage API
It was unclear why we had both of these tests when the one for
the financial year is more comprehensive - by checking data in
and beyond the specified financial year.

The only thing we lose in this file is checking multiple SMS
rates, which we will fix in the next commit when we import some
tests that are specific to variable rates.
2022-04-26 13:07:51 +01:00
Ben Thorner
86b3d60c8f Refactor billing_schemas to use list comprehension 2022-04-26 13:07:48 +01:00
Ben Thorner
a4fe11a3aa Merge pull request #3521 from alphagov/refactor-billing-tests-181934027
Small refactorings to billing APIs and tests
2022-04-26 13:05:29 +01:00
Katie Smith
d431b06244 Merge pull request #3523 from alphagov/sms-rates
Add new SMS rates for 1 May 2022 onwards
2022-04-26 11:46:35 +01:00
Katie Smith
7721cd26ca Add new SMS rates for 1 May 2022 onwards
This change can be merged before the new rates go live, because they
won't be used until the start date.
2022-04-26 10:56:25 +01:00
Leo Hemsted
46558b4577 Merge pull request #3522 from alphagov/disable-redis-enabled
remove `REDIS_ENABLED` flag from creds
2022-04-22 13:04:27 +01:00
Leo Hemsted
ae896c9880 remove REDIS_ENABLED flag from creds
you can still use this flag locally but we have it enabled for all
environments and it doesn't need to be toggleable from credentials as it
isn't a secret value.

If we wish to turn redis off for a specific environment we can create a
PR to change the config.
2022-04-22 12:05:19 +01:00
Ben Thorner
8e74280e84 Remove test file to match file under test
This makes it easier to see at a glance that this file is testing
the endpoints vs. lower level code.
2022-04-21 18:36:36 +01:00
Ben Thorner
81063ba77a Remove redundant URLs for billing API endpoints
These aren't used in Admin.
2022-04-21 15:43:03 +01:00
Ben Thorner
16133a5d4f Use admin_request consistently in billing tests
This avoids a bunch of boilerplate and makes it easier to see what
is being passed in the request.

Note that the "invalid_schema" test is slightly different because
it was previously passing an invalid JSON object - "{}" - instead
of a valid JSON but invalid by the schema - "\"{}\"". The new test
seems more valuable than the old one.
2022-04-21 15:43:01 +01:00
Sakis
b4ffcac353 Merge pull request #3519 from alphagov/custom-prometheus-prep
Use our own fork of gds_metrics_python and add shared auth token
2022-04-21 09:27:13 +01:00
sakisv
0a24b57008 Use our own fork of gds_metrics_python and add shared auth token
This will allow both prometheis (the shared one and our own) to scrape
the /metrics endpoint, each with their own authentication
2022-04-20 19:28:07 +03:00
Leo Hemsted
072c3b2079 Merge pull request #3517 from alphagov/paas-redis
bind to notify-redis automatically
2022-04-20 11:53:19 +01:00
Leo Hemsted
0457850fc0 Remove redundant conditional for CF Redis
This is now used in all environments and we've removed support for
non-CF Redis.
2022-04-20 11:41:33 +01:00
Leo Hemsted
bf083b28aa bind to notify-redis automatically
this ensures all apps are bound to redis (for example any new worker
types)
2022-04-20 11:33:27 +01:00
Ben Thorner
f67f5d987d Merge pull request #3514 from alphagov/remove-redundant-cf-code
Remove redundant CloudFoundry config code
2022-04-20 11:25:13 +01:00
Pea Tyczynska
a40e3897f0 Merge pull request #3511 from alphagov/move-nhs-orgs-to-nhs-branding
Move existing NHS orgs without branding onto NHS branding
2022-04-20 10:22:17 +01:00
Pea Tyczynska
61b6e45da5 Merge pull request #3510 from alphagov/nhs_branding_default_for_nhs_org
When creating a new NHS org, set default email branding to NHS
2022-04-19 15:33:03 +01:00
Katie Smith
9435dfc385 Merge pull request #3512 from alphagov/bump-json-schemas
Bump jsonschema package from 3.2.0 to 4.4.0
2022-04-19 14:34:39 +01:00
Katie Smith
3b7bc7c727 Merge pull request #3516 from alphagov/bump-bs4
Update beautifulsoup4 to 4.11.1
2022-04-19 14:34:29 +01:00
Katie Smith
9a249dc530 Use jsonschema[format] instead of jsonschema
`jsonschema[format]` includes all the formatting dependencies of
jsonschema, meaning that we don't have to specify `rfc3339-validator`
and `rfc3987` ourselves in the requirements.in file. This also has the
benefit of meaning that if the underlying formatting packages of
jsonschema change, we will be covered and won't accidentally miss the
fact that we need to change a package.
2022-04-19 13:53:06 +01:00
Pea Tyczynska
124562b50a Refactor creating nhs branding in tests into a fixture 2022-04-19 12:25:17 +01:00
Pea Tyczynska
769b71cdc0 When updating org type to NHS type also update email branding if none set 2022-04-19 12:07:27 +01:00
Pea Tyczynska
7da0533276 Update migrations/versions/0368_move_orgs_to_nhs_branding_.py
Co-authored-by: Ben Thorner <benthorner@users.noreply.github.com>
2022-04-19 11:53:54 +01:00
Katie Smith
ec95163175 Update beautifulsoup4 to 4.11.1
`charset-normalizer` is now used by default if installed instead of
`chardet` (https://pyup.io/changelogs/beautifulsoup4/#4.11.0). We do
have `charset-normalizer` installed because it's a subdependency of the
requests library, so it is being used.

This caused the `test_content_too_long_returns_400` to fail since it
now thought that the encoding of `ŵ` is `{'encoding': 'Big5',
'language': 'Chinese', 'confidence': 1.0}`.

There are two options for fixing this
- change the test content so that it doesn't just contain a single
  letter - the docs state that you shouldn't run character detection on
  very tiny content
- add `chardet` as a requirement, so that the code functions exactly the
  same as before

I've chose the first option, since this avoids adding a dependency and
we should never have messages consisting of a single character.
2022-04-14 16:48:32 +01:00
Katie Smith
187e87c792 Remove one of our own jsonschema date-time formatters
We have three different ways of checking the formats of datetimes.
1. The built-in way that comes with the jsonschema package ("date-time")
2. A new way we added for broadcasts ("datetime") 61a5730596
3. An old way we defined in
   "/tests/app/public_contracts/schemas/v0/definitions.json"

In order to simplify things and make it clearer how datetimes are being
validated, this replaces the few places where we were using option 3 with option 1
instead. Option 3 was only being used to validate code that is no longer
used, the initial version of the API.
2022-04-14 14:47:45 +01:00
Katie Smith
5feb38f50a Bump jsonschema from 3.2.0 to 4.4.0
The big breaking change for our code (not mentioned in the changelog) is
that the built-in validator for the `date-time` format now requires the
`rfc3339-validator` package instead of the `strict-rfc3339` package.
This updates the requirements file to use `rfc3339-validator`. Without
this change, wrong `date-time` formats would always silently pass validation.
2022-04-14 14:47:42 +01:00
Katie Smith
b440f3f904 Use Draft-07 and Draft7Validator everywhere
We were using the Draft4Validator in one place, so this updates it to
the Draft7Validator instead.

The schemas were mostly using draft 4 of the JSON schema, though there
were a couple of schemas that were already of version 7. This updates
them all to version 7, which is the latest version fully supported by
the jsonschema Python package. There are some breaking changes in the
newer version of the schema, but I could not see anywhere would these
affect us. Some of these schemas were not valid in version 4, but are
now valid in version 7 because `"required": []` was not valid in earlier
versions.
2022-04-14 14:46:10 +01:00
Katie Smith
f17e01c90a Merge pull request #3515 from alphagov/bump-straightforward-dependencies
Bump straightforward dependencies
2022-04-14 14:45:40 +01:00
Katie Smith
f6f6b81e91 Update cachetools from 4.2.1 to 5.0.0
There are breaking changes in the latest version, but these should not
affect our code.
2022-04-14 14:17:41 +01:00
Katie Smith
f4a4dd8822 Update sqlalchemy from 1.4.32 to 1.4.35 2022-04-14 13:46:19 +01:00
Katie Smith
857e7c1ce1 Update prometheus-client from 0.10.1 to 0.14.1 2022-04-14 13:39:31 +01:00
Katie Smith
667d505b5d Update flask-bcrypt from 0.7.1 to 1.0.1
There's no changelog for this, but I've looked through all the commits
and can't see any reason why this needed a major version bump or
anything that should cause us issues.
2022-04-14 13:15:36 +01:00
Katie Smith
1f705f3c29 Update flask from 2.1.0 to 2.1.1 2022-04-14 10:17:20 +01:00
Katie Smith
0cd06dba62 Update celery[sqs] from 5.2.3 to 5.2.6. 2022-04-14 10:13:30 +01:00
Katie Smith
c3829da864 Bump all test dependencies 2022-04-14 09:07:39 +01:00
Ben Thorner
95c5f0c079 Remove redundant CloudFoundry config code
These env vars can be set directly in the manifest, like we do for
Template Preview [^1].

[^1]: c08036189b/manifest.yml.j2 (L23-L26)
2022-04-13 14:46:52 +01:00
Sakis
153ffd52c4 Merge pull request #3506 from alphagov/add-internal-routes
Add internal routes for api and api-sms-receipts
2022-04-13 10:16:26 +01:00
sakisv
d6b78e6373 Add internal routes for api and api-sms-receipts
These routes will be used by prometheus to scrape the `/metrics` endpoint.

Currently:

The shared prometheus scrapes the `/metrics` endpoint using
the public routes.

The `/metrics` endpoint is provided by the [gds_metrics_python][] which
comes with [bearer-token authentication][] where the token is expected
to be equal to the paas app id.

Each app is configured as a separate target in the shared prometheus
with its app id configured as a GET parameter (e.g.
http://notify-api-production.cloudapps.digital/metrics?cf_app_guid=69c87503-6b53-4c35-XXXX-XXXXXXXXXXXX&cf_app_instance=69c87503-6b53-4c35-XXXX-XXXXXXXXXXXX%3A1&cf_app_instance_index=1)

Each scrape request goes through an nginx proxy which retrieves this
argument from the query string and sets it as a header [[source][]]. This way it
passes the authentication and also is able to instruct the gorouter to
target a specific instance of the app.

In the future:

Since we're moving away from the shared prometheus and towards an
approach where we [run our own prometheus on PaaS][] we can skip the
need for having an nginx proxy and use the internal routes instead, and
have a [preshared-token][] for authentication if we need to.

[gds_metrics_python]: https://github.com/Crown-Commercial-Service/gds_metrics_python
[bearer-token authentication]: https://github.com/Crown-Commercial-Service/gds_metrics_python/blob/master/gds_metrics/__init__.py#L47-L52
[source]: https://github.com/alphagov/prometheus-aws-configuration-beta/blob/master/terraform/modules/prom-ec2/prometheus/cloud.conf#L111-L123
[run our own prometheus on PaaS]: https://github.com/alphagov/notifications-cf-monitoring/pull/1
[preshared-token]: https://github.com/Crown-Commercial-Service/gds_metrics_python/pull/18
2022-04-13 11:01:24 +03:00
Pea Tyczynska
3777358287 Move existing nhs orgs without branding onto nhs branding
This is done to make self-service branding easier to implement,
and also because NHS branding makes much more sense for services
in those orgs than GOV.UK branding.
2022-04-12 18:28:55 +01:00
Pea Tyczynska
b1ed722252 When creating a new NHS org, set default email branding to NHS
This is more appropriate default for that org than gov.uk branding
and will help us with our work to make setting the branding more
self-service.
2022-04-12 17:24:32 +01:00
Ben Thorner
8c7ad16452 Merge pull request #3503 from alphagov/allow-repeat-send-letter
Don't error sending a letter that's sent already
2022-04-12 16:04:54 +01:00
Ben Thorner
413c6c4c26 Move check for existing letter earlier in endpoint
In response to: [^1].

[^1]: https://github.com/alphagov/notifications-api/pull/3503#discussion_r848426047
2022-04-12 15:51:06 +01:00
Leo Hemsted
91200a2088 Merge pull request #3502 from alphagov/provider-report
add new daily sms provider volume report
2022-04-12 15:48:24 +01:00
Ben Thorner
a5e0fd6104 Merge pull request #3508 from alphagov/redis-ssl-181796569
Prepare to switch to Redis on PaaS
2022-04-12 15:47:46 +01:00
Ben Thorner
5eeb74b267 Merge pull request #3509 from alphagov/remove-redundant-log-181665654
Remove redundant ternary on SMS client FROM_NUMBER
2022-04-12 15:43:33 +01:00
Ben Thorner
29fffc406c Merge pull request #3507 from alphagov/bump-utils-55-1-4
Bump utils to 55.1.4 (no changes)
2022-04-12 15:43:09 +01:00
Ben Thorner
44d90b0a4f Remove redundant ternary on SMS client FROM_NUMBER
Logs over the past 14 days confirm we never call this code with
None as the sender, so it's safe to remove the ternary.
2022-04-12 14:59:21 +01:00