How to run tests

Choose your language:  


The standard and recommended way to run tests in Python is from the terminal using pytest.

Required structure for running tests from the command line

This approach works when you have:

1. Test files that import your analysis code

Your test files need to import the functions, classes, or modules you want to test. This can be done in several ways depending on your project structure:

# If your code is packaged
from waitingtimes.patient_analysis import summary_stats

# If your code is in a local file in the same directory
from patient_analysis import summary_stats

# If your code is in a subdirectory
from src.analysis import summary_stats

2. Test files following the test_ naming convention

Pytest automatically discovers and runs tests when your test files start with test_. It is also typical (but not mandatory) to store test files in a folder called tests/. For example:

tests/
├── test_back.py
├── test_functional.py
└── test_unit.py

Common pytest commands

Once your project follows these conventions, you can run tests from a terminal, PowerShell, or command prompt (depending on your OS). Common commands include:

# Run all tests in the current directory and subdirectories
pytest

# Run all tests in the tests/ directory
pytest tests/

# Run tests from a specific file
pytest tests/test_filename.py

# Run a specific test function
pytest tests/test_filename.py::test_function_name

The standard and recommended way to run tests in R is from the R console, running them either via devtools or directly with testthat.

Example: package structure

When your code is structured as a package, your project might look like:

project/
├── data/
│   └── patient_data.csv
├── pyproject.toml
├── README.md
├── src/
│   └── waitingtimes/
│       ├── __init__.py
│       └── patient_analysis.py
└── tests/
    ├── test_intro_simple.py
    └── ...

The video below demonstrates running tests on a package structured this way. In the video we:

  • Use tree to display the file structure (matches that shown above).
  • Activate our environment: conda activate hdruk_tests (which includes pytest).
  • Install our local package (if not already installed): pip install -e ..
  • Verify the installation with conda list, confirming our waitingtimes package appears.
  • Show you the test file (nano tests/test_intro_simple.py) (matches the one from the previous page on writing basic tests).
  • Run the test with pytest tests/test_intro_simple.py - and it passes! 🎉

In an R package, tests will be automatically discovered and run if they follow the conventions of being:

  1. Stored within tests/testthat/.
  2. In R files starting with test_ or test-.

For example, your project might look like:

project/
├── inst/
│   └── extdata/
│       └── patient_data.csv
├── man/
│   └── ...
├── R/
│   └── patient_analysis.R
├── tests/
│   └── testthat/
│       ├── test_intro_simple.R
│       └── ...
├── .Rbuildignore
├── DESCRIPTION
├── LICENSE
├── LICENSE.md
├── NAMESPACE
└── README.md

You can either run the tests with devtools or testthat - common commands include:

# Run all tests in the package
devtools::test()

# Run all tests in the named file
testthat::test_file("tests/testthat/test-patient_analysis.R")

# Run all tests in that directory
testthat::test_dir("tests/testthat")

In the video below, we have an example of a research project structured as an our R project.

  • We highlight that the renv is active (Project '~/Documents/stars/hdruk_tests' loaded. [renv 1.1.5]).
  • We show the test file we will be running (test_intro_simple.R).
  • We run devtools::test() and can see that all tests pass.

If you’re unfamiliar with how to set up a package, check out our tutorial on packaging your research project.

Example: Non-package structure

You don’t need a full package to use pytest. When your code exists in .py files but isn’t packaged, you can still run tests from the terminal. Your project might, for example, look like:

project/
├── patient_analysis.py
└── test_intro_simple.py

In the video below, we:

  • Use tree to display the file structure - there is just a .py file with the summary_stats() function and test_intro_simple.py file alongside it.
  • View the test file with nano test_intro_simple.py, showing it imports locally since files are in the same folder.
  • Run the test with pytest, since the file follows the test_ naming pattern.

You don’t need a full package to use testthat. When your code exists in .R files but isn’t packaged, you can still run tests from the R console. You project might, for example, look like:

project/
├── patient_analysis.R
└── test_intro_simple.R

As your test follows of the convention of starting test_, it can still be detected by testthat and run by testthat:test_dir() or testthat::test_file().

Note: You cannot use devtools::test() with this approach.

In the video below, we:

  • Open the test_intro_simple.R file, showing how it imports the local function.
  • Run testthat::test_dir("."), which finds and runs our test.

Alternative options

While running tests from the command line is the recommended approach, there are alternative tools for specific use cases.

testbook

testbook allows you to test code within Jupyter notebooks:

# Install: pip install testbook
from testbook import testbook

@testbook('my_notebook.ipynb', execute=True)
def test_notebook_function(tb):
    func = tb.ref("add")
    assert func(2, 3) == 5

ipytest

ipytest lets you run pytest directly inside Jupyter notebook cells:

# Install: pip install ipytest
import ipytest
ipytest.autoconfig()

# In another cell
def add(a, b):
    return a + b

# In a test cell
%%ipytest
def test_add():
    assert add(2, 3) == 5

While running tests from the R console is the recommended approach, it is also possible to put tests directly in the same .R file as your code and just run the script.

For example:

# add.R

add <- function(a, b) {
  a + b
}

library(testthat)

test_that("add works correctly", {
  expect_equal(add(2, 3), 5)
  expect_equal(add(-1, 1), 0)
})
Test passed with 2 successes 🎉.

Why command line testing is better

Running tests from the terminal with code organised into separate files is the preferred approach because:

  • Test code is separate from analysis code, making both easier to understand and maintain.

  • Tests can be run automatically in continuous integration (CI) pipelines, before commits, or on schedule.

  • As your project grows, you can easily add more test files and organise them logically without cluttering notebooks or scripts.