Tests

Tests make development easier for both veteran project contributors and newcomers alike. Most projects use the unittest framework for tests so you should familiarize yourself with this framework.

Writing tests can be a great way to get involved with a project. It’s an opportunity to get familiar with the codebase and the code submission and review process. Check the project’s code coverage and write a test for a piece of code missing coverage!

Patches should be accompanied by one or more tests to demonstrate the feature or bugfix works. This makes the review process much easier since it allows the reviewer to run your code with very little effort, and it lets developers know when they break your code.

Test Organization

Having a standard test layout makes it easy to find tests. When adding new tests, follow the following guidelines:

  1. Each module in the application should have a corresponding test module. These modules should be organized in the test package to mirror the package they test. That is, if the package contains the <package>/server/push.py module, the test module should be in a module called <test_root>/server/test_push.py.

  2. Within each test module, follow the unittest code organization guidelines.

  3. Include documentation blocks for each test case that explain the goal of the test.

  4. Avoid using mock unless absolutely necessary. It’s easy to write tests using mock that only assert that mock works as expected. When testing code that makes HTTP requests, consider using vcrpy.

You may find projects that do not follow this test layout. In those cases, consider re-organizing the tests to follow the layout described here and follow the established conventions for that project until that happens.

Test Runners

Projects should include a way to run the tests with ease locally and the steps to run the tests should be documented. This should be the same way the continuous integration (Jenkins, Zuul CI, etc.) tool runs the tests.

There are many test runners available that can discover unittest based tests. These include:

Projects should choose whichever runner best suits them.

You may find projects using the nose test runner. nose is in maintenance mode and, according to their documentation, will likely cease without a new maintainer. They recommend using unittest, pytest, or nose2.

Tox

Tox is an easy way to run your project’s tests (using a Python test runner) using multiple Python interpreters. It also allows you to define arbitrary test environments, so it’s an excellent place to run the code style tests and to ensure the project’s documentation builds without errors or warnings.

You can find an example in our CookieCutter template’s tox.ini. It runs the test suite in various versions of Python, reports the coverage, runs the pre-commit checks, makes sure we’re not depending on software with forbidden licenses, and builds the documentation with the warnings treated as errors Sphinx flag enabled.

Coverage

coverage is a good way to collect test coverage statistics. pytest has a pytest-cov plugin that integrates with coverage. Diff-Cover can be used to ensure that all lines edited in a patch have coverage.

It’s possible (and recommended) to have the test suite fail if the coverage is below 100%. See the template’s pyproject.toml file for an example.

An alternative to having 100% coverage is to forbid new changes from bringing the coverage down. This can be checked by diff-cover with the following tox.ini snippet:

[tox]
envlist = pyXX...,diff-cover,...
# If the user is missing an interpreter, don't fail
skip_missing_interpreters = True

[testenv:diff-cover]
deps =
    diff-cover
commands =
    diff-cover coverage.xml --compare-branch=origin/develop --fail-under=100

You must then make sure that the pytest command line contains --cov-report xml to produce the coverage.xml file that diff-cover will analyze.

New projects should enforce 100% test coverage. Existing projects should ensure test coverage does not drop to accept a pull request and should increase the minimum test coverage until it is 100%.

coverage has great exclusion support, so you can exclude individual lines, conditional branches, functions, classes, and whole source files from your coverage report. If you have code that doesn’t make sense to have tests for, you can exclude it from your coverage report. Remember to leave a comment explaining why it’s excluded!

Licenses

The liccheck checker can verify that every dependency in your project has an acceptable license. The dependencies are checked recursively.

The licenses are validated against a set of acceptable licenses that you define in a file called .license_strategy.ini in your project directory. To avoid having to maintain and sync the same file in every project, a shared file is available in the shared repo.

The CookieCutter template contains a shell script that will download the shared file and check the dependencies' licenses against it.

If you want to add a license to the list, feel free to open a Pull Request against the shared repo.

You can automate the license check by running the shell script in Tox.

Security

The bandit checker is designed to find common security issues in Python code.

You can add it to the tests run by Tox by adding the following snippet to your tox.ini file:

[testenv:bandit]
deps = bandit
commands =
    bandit -r your_project/ -x your_project/tests/ -ll

Remember to add bandit to your Tox envlist.

Note that Ruff is also capable of running security checks equivalent to Bandit’s. Make sure you are selecting the S category in pyproject.toml.

Want to help? Learn how to contribute to Fedora Docs ›