notifications-admin/docs/end_to_end_tests.md

Working with End-to-End Tests

End-to-End (E2E) tests are an important part of a assessing the overall integrity and stability of a system. They are a part of the overall test suite and serve the function of simulating a user working through the application. By having comprehensive E2E tests in place, we can instill higher confidence that future changes and refactorings won't negatively impact any user experience or break existing functionality.

The US Notify project leverages pytest for its existing test suite (at least on the Python side of things) and is now leveraging Playwright for Python along with its pytest plugin for the E2E tests.

Getting Started

To work with the E2E tests in US Notify, you need to make sure you have all of the necessary components installed. The quick and easy way to do this is to use the Makefile as you did for the initial project setup. In fact, if you've already done this, you are already set to go! If not, then run the bootstrap command in your shell:

make bootstrap

This takes care of installing all of your dependencies, including those now needed for Playwright.

If you run into certificate errors at the playwright install step, try doing this:

1. Run brew --prefix to see Homebrew's root directory

2. Create or modify the local .env file in the project and add this line:

   NODE_EXTRA_CA_CERTS=/CHANGE-TO-HOMEBREW-INSTALL-PATH/etc/ca-certificates/cert.pem

   Make sure to change CHANGE-TO-HOMEBREW-INSTALL-PATH to the path given by brew --prefix in the step above. For example, if brew --prefix gave /opt/homebrew as output, then the line would look like this:

   NODE_EXTRA_CA_CERTS=/opt/homebrew/etc/ca-certificates/cert.pem

3. Save the changes to the .env file

4. Run make bootstrap again

Manual Installation

If you need to install things separately, you'll still need to make sure your environment is set up and configured as outlined in the README.

At your shell in the project root folder, run the following commands:

poetry install pytest-playwright --dev
poetry run playwright install --with-deps

This will install Playwright and its pytest plugin, then the additional dependencies that Playwright requires.

See more details on the Playwright for Python Installation page.

Local Configuration

In order to run the E2E tests successfully on your local machine, you'll also need to make sure you have a .env file in the root project folder and that it has at least these environment variables set in it:

# E2E Test Configuration - only set for the Admin site.
NOTIFY_E2E_TEST_URI
NOTIFY_E2E_TEST_EMAIL
NOTIFY_E2E_TEST_PASSWORD
NOTIFY_E2E_AUTH_STATE_PATH

This file is not checked into source control and is configured to be ignored in the project's .gitignore file; please be careful that it is not committed to the repo and pushed!

Running E2E Tests Locally

To run the E2E tests on your local machine, type this command in your shell at the project root directory:

make e2e-test

You should see pytest start producing output and the existing E2E tests run in multiple headless browsers.

How to Create and Maintain E2E Tests

All of the E2E tests are found in the tests/end_to_end folder and are written as pytest scripts using Playwright's Python Framework.

Inside the tests/end_to_end folder you'll see a conftest.py file, which is similar to the one found in the root tests folder but is specific to the E2E tests.

There a few fixtures defined in here, but the two most important at this time are these:

• end_to_end_context: A Playwright context object needed to interact with a browser instance.
• authenticated_page: A Playwright page object that has gone through the sign in process the E2E user is authenticated.

In short, if you're starting a test from scratch and testing pages that do not require authentication, you'll start with the end_to_end_context fixture and work from there.

Any test that requires you to be authenticated, you'll start with the authenticated_page object as that'll have taken care of getting everything set for you and logged into the site with the E2E test user.

Creating a new test file

If you want to create a new test file to help organize tests (a great idea!), it will be handy to import the Playwright expect and set the base URL/URI for yourself, like this:

from playwright.sync_api import expect

E2E_TEST_URI = os.getenv("NOTIFY_E2E_TEST_URI")

By importing Playwright's expect object for tests and setting something like E2E_TEST_URI for yourself, it will make writing tests much easier.

Using the fixtures

To use the authenticated_page or end_to_end_context fixtures, you start by defining a test function and then passing in the fixture you need as a positional argument. This works the same as the other functions defined to create a test for pytest.

For example, the test for the landing page starts with this:

def test_landing_page(end_to_end_context):
    # Open a new page and go to the site.
    page = end_to_end_context.browser.new_page()
    page.goto(f"{E2E_TEST_URI}/")

    # Check to make sure that we've arrived at the next page.
    page.wait_for_load_state("domcontentloaded")
    ...

Note the passing in of the end_to_end_context fixture - there is no need to import this or anything, just pass it into the function. pytest takes care of everything else for you.

The second line that defines a page variable is a convenience, since you'll be referencing the page object a lot. This is recommended to help keep tests readable while keeping fixture names descriptive.

If you need to test an authenticate page, such as the accounts page, use the authenticated_page fixture instead, like so:

def test_add_new_service_workflow(authenticated_page):
    page = authenticated_page
    ...

Again, it's helpful to assign the fixture to a page variable for easy reference throughout the test.

Lastly, if you need want access to the Playwright context object that is used behind the page fixtures, you can reference it directly as well using the end_to_end_context fixture:

def test_add_new_service_workflow(authenticated_page, end_to_end_context):
    page = authenticated_page

    # Prepare for adding a new service later in the test.
    current_date_time = datetime.datetime.now()
    new_service_name = "E2E Federal Test Service {now} - {browser_type}".format(
        now=current_date_time.strftime("%m/%d/%Y %H:%M:%S"),
        browser_type=end_to_end_context.browser.browser_type.name,
    )
    ...

In this example, I've used the context to get to the browser object itself to get the name of the browser for test data.

Maintaining E2E Tests with GitHub

The E2E tests are configured to run as a separate GitHub action as a part of our other checks found in .github/workflows/checks.yml.

The E2E tests are not run as a part of the regular unit test suite; if you look at the Makefile you'll see that the tests are two separate commands, with the E2E tests configured separately.

This is done for a couple of reasons:

• Keeps unit tests isolated from the E2E tests
• Allows us to configure E2E tests separately

The environment variables are managed as a part of the GitHub environment and repository settings.

E2E Environment Variable Management

These are the E2E test environment variables that must be set:

NOTIFY_E2E_TEST_URI
NOTIFY_E2E_TEST_EMAIL
NOTIFY_E2E_TEST_PASSWORD
NOTIFY_E2E_AUTH_STATE_PATH

These are only set for the Admin site in GitHub, but must be set for both GitHub Actions and Dependabot for the same reason as the MFA environment variables.