Test coverage

Choose your language:  

Coverage refers to the percentage of your code that is executed when you run your tests. It can help you spot parts of your code that are not included in any tests.

pytest-cov

The pytest-cov package can be used to run coverage calculations easily alongside pytest. You can install it from PyPI or conda:

pip install pytest-cov

conda install pytest-cov

To calculate coverage, you can then simply run tests with the --cov flag:

pytest --cov

Running pytest --cov on our example

NoteTest output

============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /__w/hdruk_tests/hdruk_tests/examples/python_package
configfile: pyproject.toml
plugins: cov-7.0.0
collected 13 items

../examples/python_package/tests/test_back.py .                          [  7%]
../examples/python_package/tests/test_functional.py ...                  [ 30%]
../examples/python_package/tests/test_intro_parametrised.py ..           [ 46%]
../examples/python_package/tests/test_intro_simple.py .                  [ 53%]
../examples/python_package/tests/test_unit.py ......                     [100%]

================================ tests coverage ================================
_______________ coverage: platform linux, python 3.12.12-final-0 _______________

Name                                                                      Stmts   Miss  Cover
---------------------------------------------------------------------------------------------
/workspace/examples/python_package/src/waitingtimes/__init__.py               1      0   100%
/workspace/examples/python_package/src/waitingtimes/patient_analysis.py      30      1    97%
---------------------------------------------------------------------------------------------
TOTAL                                                                        31      1    97%
============================== 13 passed in 1.91s ==============================
<ExitCode.OK: 0>

The coverage results are under the banner:

================================ tests coverage ================================

You can see we get nearly 100% coverage. But what does this actually mean?

Tools for calculating test coverage in R

If your research is structured as a package, then you can use devtools to calculate coverage:

devtools::test_coverage()

If not structured as a package, you can use covr’s file_coverage() function - for example:

covr::file_coverage(
  source_files = c("patient_analysis.R"),
  test_files = c("test_unit.R", "test_functional.R", "test_back.R")
)

Calculating coverage for our example

devtools::load_all("../examples/r_package")

devtools::test_coverage("../examples/r_package")

ℹ Computing test coverage for waitingtimes

You can see we get 100% coverage. But what does this actually mean?

Interpreting coverage

Coverage is telling you whether code was executed during testing - but not necessarily whether it has been tested well. A function could run as part of another test without its results or behaviour being properly checked by assertions.

Coverage tells you what code ran, not whether it worked correctly.

It’s mostly useful for finding code that isn’t covered by tests at all. Having parts of your code with no/low coverage means:

• They’re not imported or run by any tests.
• They’re only used in rare branches or failure conditions.
• They were added recently but have not yet been incorporated into tests.

Rather than try to achieve 100% coverage, you should aim to meaningfully test all your code: every important path, decision and behaviour should be tested at least once.