Scenario and sensitivity analysis

Learning objectives:

  • Recognise the importance of sharing your scenario and sensitivity analysis code.
  • Learn how to run a combination of scenarios.
  • Understand differences between scenario and sensitivity analysis.

Relevant reproducibility guidelines:

  • STARS Reproducibility Recommendations (⭐): Provide code for all scenarios and sensitivity analyses.
  • STARS Reproducibility Recommendations: Save outputs to a file.
  • STARS Reproducibility Recommendations: Avoid excessive output files.
  • STARS Reproducibility Recommendations: Address large file sizes.

Pre-reading:

This page continues on from: Parallel processing.

Entity generation → Entity processing → Initialisation bias → Performance measures → Replications → Parallel processing → Scenario and sensitivity analysis

Required packages:

These should be available from environment setup in the “Test yourself” section of Environments.

import itertools

from IPython.display import HTML
from itables import to_html_datatable
import numpy as np
import pandas as pd
import scipy.stats as st
import simpy
from sim_tools.distributions import Exponential
library(dplyr)
library(kableExtra)
library(knitr)
library(simmer)
library(tidyr)

Running and sharing scenarios

When creating a DES, you won’t just run one model - you run scenarios, which are different parameter set-ups. These are used to test how outcomes change under varying conditions.

When running scenarios, there are two important things to remember:

1. Run scenarios programmatically.

Don’t duplicate whole scripts just to test different set-ups. The idea is to build the model with functions and classes so you only need to change the parameters and re-run. That’s why we guide you through building the model this way in this book.

2. Share your scenario code.

You should share code and data for every scenario - not just the base case.

In Heather et al. (2025), six out of seven studies with scenarios lacked scenario code. This made it challenging and time-consuming to reproduce their results, as descriptions of the scenarios in the articles were often ambiguous and key parameters unclear or missing.

Simple scenarios

This code runs the simulation for each value of number_of_doctors from 3 to 6.

The returned dataframe contains the results from each of the five runs from the four scenarios.

results = []
# Loop through doctor counts 3-6, running the simulation
for i in range(3, 7):
    param = Parameters(number_of_doctors=i)
    runner = Runner(param=param)
    result = runner.run_reps()["run"]
    # Add scenario information to the results, then save to list
    result["number_of_doctors"] = i
    results.append(result)

# Combine results into a single dataframe
scenario_results = pd.concat(results, ignore_index=True)

# Show as interactive table
HTML(to_html_datatable(scenario_results))
Loading ITables v2.5.2 from the internet... (need help?) 
results <- list()
# Loop through doctor counts 3-6, running the simulation
for (i in 3L:6L) {
  param <- create_params(number_of_doctors = i)
  result <- runner(param = param)[["run_results"]]
  # Add scenario information to the results, then save to list
  result["number_of_doctors"] <- i
  results[[length(results) + 1L]] <- result
}

# Combine results into a single dataframe
scenario_results <- do.call(rbind, results)

# Show as interactive table
kable(scenario_results) |> scroll_box(height = "400px")
replication arrivals mean_wait_time_doctor mean_time_with_doctor utilisation_doctor mean_queue_length_doctor mean_time_in_system mean_patients_in_system number_of_doctors
1 9 1.1561321 12.600360 0.7545527 0.3739683 6.972457 2.4937655 3
2 10 0.2185845 15.462225 0.8137894 0.7190226 7.019465 3.0715488 3
3 8 1.6612341 11.704709 0.8144131 0.2671798 10.033307 1.7999155 3
4 4 1.3862671 8.268402 0.5633356 0.9223766 8.714268 0.8490077 3
5 7 1.1383003 8.211288 0.8027837 0.1753468 8.159880 1.5179933 3
1 9 0.0000000 14.361497 0.5564580 0.0000000 5.909446 2.0694053 4
2 8 0.0000000 10.902873 0.5209432 0.0000000 5.040122 1.6635569 4
3 8 0.1002196 12.491221 0.7258143 0.0000000 8.233837 1.9542346 4
4 5 0.0000000 7.220270 0.3280956 0.0000000 8.605687 0.8995180 4
5 5 0.0000000 7.764737 0.2639238 0.0000000 7.764737 0.9782088 4
1 9 0.0000000 14.361497 0.4451664 0.0000000 5.909446 2.0694053 5
2 8 0.0000000 10.902873 0.4167546 0.0000000 5.040122 1.6635569 5
3 8 0.0000000 13.200172 0.5846602 0.0000000 8.233837 1.9542346 5
4 5 0.0000000 7.220270 0.2624765 0.0000000 8.605687 0.8995180 5
5 5 0.0000000 7.764737 0.2111390 0.0000000 7.764737 0.9782088 5
1 9 0.0000000 14.361497 0.3709720 0.0000000 5.909446 2.0694053 6
2 8 0.0000000 10.902873 0.3472955 0.0000000 5.040122 1.6635569 6
3 8 0.0000000 13.200172 0.4872168 0.0000000 8.233837 1.9542346 6
4 5 0.0000000 7.220270 0.2187304 0.0000000 8.605687 0.8995180 6
5 5 0.0000000 7.764737 0.1759492 0.0000000 7.764737 0.9782088 6

Running several scenario combinations

If you are running several scenarios, it can be useful to define a helper function like the one shown below. This example is adapted from:

“HSMA - little book of DES” from Sammi Rosser, Dan Chalk and Amy Heather 2025 (MIT Licence).

def run_scenarios(scenarios, param_factory=None):
    """
    Execute a set of scenarios and return the results from each run.

    Parameters
    ----------
    scenarios : dict
        Dictionary where key is name of parameter and value is a list with
        different values to run in scenarios.
    param_factory : callable or None, optional
        A callable that returns a new Parameters object for each scenario run.
        This can be a class (e.g., `Parameters`) or a factory function/lambda
        with preset arguments (e.g., `lambda: Parameters(number_of_runs=4)`).
        If not provided, defaults to using `Parameters()` with no arguments.

    Returns
    -------
    pandas.DataFrame
        DataFrame with results from each run of each scenario.

    Notes
    -----
    Function adapted from Rosser, Chalk and Heather 2025.
    """
    # If none provided, use Parameters
    if param_factory is None:
        param_factory = Parameters

    # Find every possible permutation of the scenarios
    all_scenarios_tuples = list(itertools.product(*scenarios.values()))

    # Convert back into dictionaries
    all_scenarios_dicts = [
        dict(zip(scenarios.keys(), p)) for p in all_scenarios_tuples
    ]

    # Preview some of the scenarios
    print(f"There are {len(all_scenarios_dicts)} scenarios. Running:")

    # Run the scenarios...
    results = []
    for index, scenario_to_run in enumerate(all_scenarios_dicts):
        print(scenario_to_run)

        # Create fresh instance of parameter class for each scenario
        param = param_factory()

        # Update parameter list with the scenario parameters
        param.scenario_name = index
        for key in scenario_to_run:
            setattr(param, key, scenario_to_run[key])

        # Perform replications
        scenario_exp = Runner(param)
        scenario_res = scenario_exp.run_reps()["run"]

        # Add scenario number and values to the results dataframe
        scenario_res["scenario"] = index
        for key in scenario_to_run:
            scenario_res[key] = scenario_to_run[key]

        # Add results from scenario to list
        results.append(scenario_res)
    return pd.concat(results)

If you are running several scenarios, it can be useful to define a helper function like the one shown below.

#' Run a set of scenarios
#'
#' @param scenarios List where key is name of parameter and value is a list of
#' different values to run in scenarios
#' @param base_list List of parameters to use as base for scenarios, which can
#' be partial (as will input to create_params() function).
#' @param verbose Boolean, whether to print messages about scenarios as run.
#'
#' @return Tibble with results from each replication for each scenario.
#' @export

run_scenarios <- function(scenarios, base_list, verbose = TRUE) {
  # Generate all permutations of the scenarios
  all_scenarios <- expand.grid(scenarios)

  # Preview the number of scenarios
  if (isTRUE(verbose)) {
    message(sprintf("There are %d scenarios.", nrow(all_scenarios)))
    message("Base parameters:")
    print(base_list)
  }

  results <- list()

  # Iterate through each scenario
  for (index in seq_len(nrow(all_scenarios))) {

    # Filter to one of the scenarios
    scenario_to_run <- all_scenarios[index, , drop = FALSE]

    # Print the scenario parameters
    formatted_scenario <- toString(
      paste0(names(scenario_to_run), " = ", scenario_to_run)
    )

    # Print the scenario currently running
    if (isTRUE(verbose)) {
      message("Scenario: ", formatted_scenario)
    }

    # Create parameter list with scenario-specific values
    s_args <- c(scenario_to_run, list(scenario_name = index))

    # Create instance of parameter class with specified base parameters
    s_param <- do.call(create_params, base_list)

    # Update parameter list with the scenario parameters
    for (name in names(s_args)) {
      s_param[[name]] <- s_args[[name]]
    }

    # Run replications for the current scenario and get processed results
    scenario_result <- runner(s_param)[["run_results"]]

    # Append scenario parameters to the results
    scenario_result[["scenario"]] <- index
    for (key in names(scenario_to_run)) {
      scenario_result[[key]] <- scenario_to_run[[key]]
    }

    # Append to results list
    results[[index]] <- scenario_result
  }
  do.call(rbind, results)
}

The function generates all possible scenario combinations from the values supplied in scenarios.

It then iterates through these, creating a fresh set of parameters for each scenario run. This ensures every scenario gets an independent set of parameters and avoids carrying state from one scenario to another.

The replication results from each scenario are saved in a list and finally combined into a single table.

Let’s run it!

# Run scenarios
scenario_results = run_scenarios(
    scenarios={"interarrival_time": [4, 5, 6, 7, 8],
               "number_of_doctors": [3, 4, 5]},
    param_factory=Parameters
)
There are 15 scenarios. Running:
{'interarrival_time': 4, 'number_of_doctors': 3}
{'interarrival_time': 4, 'number_of_doctors': 4}
{'interarrival_time': 4, 'number_of_doctors': 5}
{'interarrival_time': 5, 'number_of_doctors': 3}
{'interarrival_time': 5, 'number_of_doctors': 4}
{'interarrival_time': 5, 'number_of_doctors': 5}
{'interarrival_time': 6, 'number_of_doctors': 3}
{'interarrival_time': 6, 'number_of_doctors': 4}
{'interarrival_time': 6, 'number_of_doctors': 5}
{'interarrival_time': 7, 'number_of_doctors': 3}
{'interarrival_time': 7, 'number_of_doctors': 4}
{'interarrival_time': 7, 'number_of_doctors': 5}
{'interarrival_time': 8, 'number_of_doctors': 3}
{'interarrival_time': 8, 'number_of_doctors': 4}
{'interarrival_time': 8, 'number_of_doctors': 5}
# Save to CSV
scenario_results.to_csv(
    "scenarios_resources/python_scenario_results.csv", index=False
)

# View scenario results
HTML(to_html_datatable(scenario_results.head(200)))
Loading ITables v2.5.2 from the internet... (need help?) 
# Run scenarios
scenarios <- list(
  interarrival_time = c(4L, 5L, 6L, 7L, 8L),
  number_of_doctors = c(3L, 4L, 5L)
)
scenario_results <- run_scenarios(
  scenarios = scenarios, base_list = create_params()
)
There are 15 scenarios.
Base parameters:
$interarrival_time
[1] 5

$consultation_time
[1] 10

$number_of_doctors
[1] 3

$warm_up_period
[1] 30

$data_collection_period
[1] 40

$number_of_runs
[1] 5

$verbose
[1] FALSE
Scenario: interarrival_time = 4, number_of_doctors = 3
Scenario: interarrival_time = 5, number_of_doctors = 3
Scenario: interarrival_time = 6, number_of_doctors = 3
Scenario: interarrival_time = 7, number_of_doctors = 3
Scenario: interarrival_time = 8, number_of_doctors = 3
Scenario: interarrival_time = 4, number_of_doctors = 4
Scenario: interarrival_time = 5, number_of_doctors = 4
Scenario: interarrival_time = 6, number_of_doctors = 4
Scenario: interarrival_time = 7, number_of_doctors = 4
Scenario: interarrival_time = 8, number_of_doctors = 4
Scenario: interarrival_time = 4, number_of_doctors = 5
Scenario: interarrival_time = 5, number_of_doctors = 5
Scenario: interarrival_time = 6, number_of_doctors = 5
Scenario: interarrival_time = 7, number_of_doctors = 5
Scenario: interarrival_time = 8, number_of_doctors = 5
# Save to CSV
write.csv(scenario_results,
          file.path("scenarios_resources", "r_scenario_results.csv"))

# View scenario results
kable(scenario_results) |> scroll_box(height = "400px")
replication arrivals mean_wait_time_doctor mean_time_with_doctor utilisation_doctor mean_queue_length_doctor mean_time_in_system mean_patients_in_system scenario interarrival_time number_of_doctors
1 8 0.0000000 10.626767 0.4811191 0.0000000 5.909446 1.5730708 1 4 3
2 14 1.3489715 16.474475 0.8415543 1.4037854 6.529895 4.0585764 1 4 3
3 8 3.3185568 12.576986 0.9214641 0.6425483 14.293369 2.4214309 1 4 3
4 9 7.0789803 8.229096 1.0000000 1.9960925 14.810041 3.2462020 1 4 3
5 11 11.8431217 8.866916 1.0000000 2.9270728 16.007073 4.2730406 1 4 3
1 9 1.1561321 12.600360 0.7545527 0.3739683 6.972457 2.4937655 2 5 3
2 10 0.2185845 15.462225 0.8137894 0.7190226 7.019465 3.0715488 2 5 3
3 8 1.6612341 11.704709 0.8144131 0.2671798 10.033307 1.7999155 2 5 3
4 4 1.3862671 8.268402 0.5633356 0.9223766 8.714268 0.8490077 2 5 3
5 7 1.1383003 8.211288 0.8027837 0.1753468 8.159880 1.5179933 2 5 3
1 4 0.0000000 16.756011 0.5829234 0.0000000 7.594901 1.6885375 3 6 3
2 7 0.0000000 13.951263 0.6247704 0.0000000 9.878982 1.8347756 3 6 3
3 11 0.0000000 10.642901 0.6696266 0.0000000 5.471170 2.1742571 3 6 3
4 5 2.1255795 7.962361 0.4490071 0.2915664 10.087940 1.3321312 3 6 3
5 6 0.1698502 7.931855 0.6664404 0.0000000 7.985399 0.9107067 3 6 3
1 5 0.0000000 14.928869 0.5309547 0.0000000 8.986369 1.5562112 4 7 3
2 5 0.0000000 16.557442 0.4795871 0.0000000 9.466947 1.2655765 4 7 3
3 8 0.0000000 10.066787 0.5494173 0.0000000 3.468666 2.0459750 4 7 3
4 5 0.9021887 7.962361 0.4504371 0.3962059 8.864549 1.3634454 4 7 3
5 4 0.0000000 8.753199 0.3340787 0.0000000 7.926292 0.8053229 4 7 3
1 3 0.0000000 20.737360 0.4033890 0.0000000 8.986369 1.2630193 5 8 3
2 5 0.0000000 16.557442 0.4525148 0.0000000 9.466947 1.1847657 5 8 3
3 7 0.0000000 11.304930 0.4485586 0.0000000 3.882443 1.8976861 5 8 3
4 6 0.0000000 6.932281 0.3689745 0.0000000 4.751341 1.0837230 5 8 3
5 3 0.0000000 9.563759 0.2902283 0.0000000 7.926292 0.5974274 5 8 3
1 11 0.2873017 8.979263 0.7388989 0.0000000 5.072421 2.0511195 6 4 4
2 7 0.0000000 11.322431 0.5589778 0.0000000 5.722932 1.9471475 6 4 4
3 10 0.3279443 13.891435 0.8192434 0.0000000 11.073710 2.3708711 6 4 4
4 6 0.0000000 7.155858 0.3352238 0.0000000 6.212797 0.9218882 6 4 4
5 8 0.0000000 9.299128 0.3465470 0.0000000 7.062117 1.6075429 6 4 4
1 9 0.0000000 14.361497 0.5564580 0.0000000 5.909446 2.0694053 7 5 4
2 8 0.0000000 10.902873 0.5209432 0.0000000 5.040122 1.6635569 7 5 4
3 8 0.1002196 12.491221 0.7258143 0.0000000 8.233837 1.9542346 7 5 4
4 5 0.0000000 7.220270 0.3280956 0.0000000 8.605687 0.8995180 7 5 4
5 5 0.0000000 7.764737 0.2639238 0.0000000 7.764737 0.9782088 7 5 4
1 4 0.0000000 16.756011 0.4371926 0.0000000 7.594901 1.6885375 8 6 4
2 7 0.0000000 13.951263 0.4685778 0.0000000 9.878982 1.8347756 8 6 4
3 11 0.0000000 10.921073 0.5067605 0.0000000 5.471170 2.1742571 8 6 4
4 6 0.0000000 6.932281 0.3478921 0.0000000 6.932281 1.0897496 8 6 4
5 4 0.0000000 8.753199 0.2551105 0.0000000 7.926292 0.9878955 8 6 4
1 5 0.0000000 14.928869 0.3982160 0.0000000 8.986369 1.5562112 9 7 4
2 5 0.0000000 16.557442 0.3596903 0.0000000 9.466947 1.2655765 9 7 4
3 8 0.0000000 10.066787 0.4120630 0.0000000 3.468666 2.0459750 9 7 4
4 6 0.0000000 6.932281 0.3489646 0.0000000 6.932281 1.2656769 9 7 4
5 4 0.0000000 7.539044 0.2003486 0.0000000 5.772494 0.5339783 9 7 4
1 3 0.0000000 20.737360 0.3025417 0.0000000 8.986369 1.2630193 10 8 4
2 5 0.0000000 16.557442 0.3393861 0.0000000 9.466947 1.1847657 10 8 4
3 7 0.0000000 11.304930 0.3364190 0.0000000 3.882443 1.8976861 10 8 4
4 6 0.0000000 6.932281 0.2767308 0.0000000 4.751341 1.0837230 10 8 4
5 4 0.0000000 7.437496 0.1859374 0.0000000 7.437496 0.8162890 10 8 4
1 13 0.1711051 9.688685 0.6289244 0.0403352 5.063579 2.2831273 11 4 5
2 7 0.0000000 11.322431 0.4471823 0.0000000 5.722932 1.9471475 11 4 5
3 11 0.1017516 14.993140 0.7521643 0.0000000 11.329369 2.7976261 11 4 5
4 6 0.0000000 7.155858 0.2681790 0.0000000 6.212797 0.9218882 11 4 5
5 8 0.0000000 9.299128 0.2772376 0.0000000 7.062117 1.6075429 11 4 5
1 9 0.0000000 14.361497 0.4451664 0.0000000 5.909446 2.0694053 12 5 5
2 8 0.0000000 10.902873 0.4167546 0.0000000 5.040122 1.6635569 12 5 5
3 8 0.0000000 13.200172 0.5846602 0.0000000 8.233837 1.9542346 12 5 5
4 5 0.0000000 7.220270 0.2624765 0.0000000 8.605687 0.8995180 12 5 5
5 5 0.0000000 7.764737 0.2111390 0.0000000 7.764737 0.9782088 12 5 5
1 4 0.0000000 16.756011 0.3497540 0.0000000 7.594901 1.6885375 13 6 5
2 7 0.0000000 13.951263 0.3748622 0.0000000 9.878982 1.8347756 13 6 5
3 11 0.0000000 10.921073 0.4054084 0.0000000 5.471170 2.1742571 13 6 5
4 6 0.0000000 6.932281 0.2783137 0.0000000 6.932281 1.0897496 13 6 5
5 4 0.0000000 8.753199 0.2040884 0.0000000 7.926292 0.9878955 13 6 5
1 5 0.0000000 14.928869 0.3185728 0.0000000 8.986369 1.5562112 14 7 5
2 5 0.0000000 16.557442 0.2877523 0.0000000 9.466947 1.2655765 14 7 5
3 8 0.0000000 10.066787 0.3296504 0.0000000 3.468666 2.0459750 14 7 5
4 6 0.0000000 6.932281 0.2791717 0.0000000 6.932281 1.2656769 14 7 5
5 4 0.0000000 7.539044 0.1602788 0.0000000 5.772494 0.5339783 14 7 5
1 3 0.0000000 20.737360 0.2420334 0.0000000 8.986369 1.2630193 15 8 5
2 5 0.0000000 16.557442 0.2715089 0.0000000 9.466947 1.1847657 15 8 5
3 7 0.0000000 11.304930 0.2691352 0.0000000 3.882443 1.8976861 15 8 5
4 6 0.0000000 6.932281 0.2213847 0.0000000 4.751341 1.0837230 15 8 5
5 4 0.0000000 7.437496 0.1487499 0.0000000 7.437496 0.8162890 15 8 5


NoteSelective re-running

For simulations with very long run times, you may wish to explore the {targets} package.

With {targets}, you can set up your pipeline so that input changes (like updates to parameter files or scenario definitions) trigger the package to automatically re-run only the affected simulations and analyses, while all other results are reused from cache. This targeted re-running helps save computation time when there are many slow scenarios to run.

If you change the code inside your model function, {targets} will detect those updates and make sure all impacted analysis steps are re-run.

Note: This package hasn’t been personally tried in this project, so we can’t vouch for whether it definitely works reliably with simmer simulation models - it’s just a cool package we’ve heard about! For guides on using this, see the official {targets} manual.

Sensitivity analysis

How does sensitivity analysis differ from scenario analysis?

Scenario analysis focuses on a set of pre-defined situations which are plausible and relevant to the problem being studied. It can often involve varying multiple parameters simultaneously. The purpose is to understand how the system operates under different hypothetical scenarios.

Sensitivity analysis varies one parameter (or a small group of parameters) and assess the impact of small changes in that parameter on outcomes. The purpose is to understand how uncertainty in the inputs affects the model, and how robust the model is to variation in those inputs.

Sensitivity analysis is an important for model validation (and forms part of “experimentation validation”).

Running sensitivity analyses

Sensitivity analyses should be run programmatically, just like scenario analyses, and the code should be shared for reproducibility.

They can be run using similar code. Below is an example using the run_scenarios function to vary the interarrival_time parameter.

# Run sensitivity analysis
sensitivity_results = run_scenarios(
    scenarios={"interarrival_time": [4, 4.5, 5, 5.5, 6, 6.5, 7]},
    param_factory=Parameters
)
There are 7 scenarios. Running:
{'interarrival_time': 4}
{'interarrival_time': 4.5}
{'interarrival_time': 5}
{'interarrival_time': 5.5}
{'interarrival_time': 6}
{'interarrival_time': 6.5}
{'interarrival_time': 7}
# Save to CSV
sensitivity_results.to_csv(
    "scenarios_resources/python_sensitivity_results.csv", index=False
)

# View sensitivity results
HTML(to_html_datatable(sensitivity_results.head(200)))
Loading ITables v2.5.2 from the internet... (need help?) 
# Run sensitivity analysis
sensitivity_results <- run_scenarios(
  scenarios = list(interarrival_time = c(4L, 4.5, 5L, 5.5, 6L, 6.5, 7L)),
  base_list = create_params()
)
There are 7 scenarios.
Base parameters:
$interarrival_time
[1] 5

$consultation_time
[1] 10

$number_of_doctors
[1] 3

$warm_up_period
[1] 30

$data_collection_period
[1] 40

$number_of_runs
[1] 5

$verbose
[1] FALSE
Scenario: interarrival_time = 4
Scenario: interarrival_time = 4.5
Scenario: interarrival_time = 5
Scenario: interarrival_time = 5.5
Scenario: interarrival_time = 6
Scenario: interarrival_time = 6.5
Scenario: interarrival_time = 7
# Save to CSV
write.csv(sensitivity_results,
          file.path("scenarios_resources", "r_sensitivity_results.csv"))

# View sensitivity results
kable(sensitivity_results) |> scroll_box(height = "400px")
replication arrivals mean_wait_time_doctor mean_time_with_doctor utilisation_doctor mean_queue_length_doctor mean_time_in_system mean_patients_in_system scenario interarrival_time
1 8 0.0000000 10.626767 0.4811191 0.0000000 5.909446 1.5730708 1 4.0
2 14 1.3489715 16.474475 0.8415543 1.4037854 6.529895 4.0585764 1 4.0
3 8 3.3185568 12.576986 0.9214641 0.6425483 14.293369 2.4214309 1 4.0
4 9 7.0789803 8.229096 1.0000000 1.9960925 14.810041 3.2462020 1 4.0
5 11 11.8431217 8.866916 1.0000000 2.9270728 16.007073 4.2730406 1 4.0
1 11 4.4136200 10.968638 0.9305843 0.9874912 10.970224 3.2130228 2 4.5
2 10 4.2952348 11.628385 0.8801942 1.1333248 10.737537 3.5719202 2 4.5
3 8 6.3476572 13.246809 0.9553582 1.1892971 16.012763 3.1906477 2 4.5
4 4 1.6258165 9.999398 0.6295231 2.1705516 8.013161 0.7738957 2 4.5
5 7 1.4030508 7.662403 0.7949871 0.1776756 8.671934 1.8245624 2 4.5
1 9 1.1561321 12.600360 0.7545527 0.3739683 6.972457 2.4937655 3 5.0
2 10 0.2185845 15.462225 0.8137894 0.7190226 7.019465 3.0715488 3 5.0
3 8 1.6612341 11.704709 0.8144131 0.2671798 10.033307 1.7999155 3 5.0
4 4 1.3862671 8.268402 0.5633356 0.9223766 8.714268 0.8490077 3 5.0
5 7 1.1383003 8.211288 0.8027837 0.1753468 8.159880 1.5179933 3 5.0
1 8 0.0000000 13.183604 0.6555458 0.0736313 7.594901 2.0286140 4 5.5
2 6 0.0000000 14.204232 0.6476341 0.0000000 8.061193 1.7041228 4 5.5
3 9 1.0701721 10.484981 0.7546664 0.2840460 7.720035 2.5252492 4 5.5
4 8 1.6097542 12.349744 0.6931755 0.2616702 7.074266 2.0253240 4 5.5
5 7 0.6407407 8.647519 0.7528266 0.0000000 7.929976 1.3341535 4 5.5
1 4 0.0000000 16.756011 0.5829234 0.0000000 7.594901 1.6885375 5 6.0
2 7 0.0000000 13.951263 0.6247704 0.0000000 9.878982 1.8347756 5 6.0
3 11 0.0000000 10.642901 0.6696266 0.0000000 5.471170 2.1742571 5 6.0
4 5 2.1255795 7.962361 0.4490071 0.2915664 10.087940 1.3321312 5 6.0
5 6 0.1698502 7.931855 0.6664404 0.0000000 7.985399 0.9107067 5 6.0
1 5 0.0000000 14.928869 0.5671720 0.0000000 7.601250 1.5875062 6 6.5
2 6 0.0000000 15.719055 0.5351976 0.0000000 12.100828 1.5937329 6 6.5
3 8 0.0000000 10.066787 0.5907619 0.0000000 5.471170 2.0509018 6 6.5
4 5 1.5138841 7.962361 0.4497221 0.3399041 9.476245 1.3465966 6 6.5
5 5 0.0046527 13.128797 0.6221594 0.0000000 13.133450 1.7319608 6 6.5
1 5 0.0000000 14.928869 0.5309547 0.0000000 8.986369 1.5562112 7 7.0
2 5 0.0000000 16.557442 0.4795871 0.0000000 9.466947 1.2655765 7 7.0
3 8 0.0000000 10.066787 0.5494173 0.0000000 3.468666 2.0459750 7 7.0
4 5 0.9021887 7.962361 0.4504371 0.3962059 8.864549 1.3634454 7 7.0
5 4 0.0000000 8.753199 0.3340787 0.0000000 7.926292 0.8053229 7 7.0

Saving results

Saving your simulation results to file is important for reproducility, as it allows others to verify your findings and generate consistent (or new) figures and analyses, even if they can’t re-run your simulation. This practice is transparent, providing a clear record of what you found, and it is valuable for you as well, ensuring you always know exactly what results you obtained, and can regenerate your own tables and figures from the results.

However, there are two key things to keep in mind:

1. Number of files. Running many scenarios or replications can easily lead to an explosion of output files. Do not save each scenario or run as a separate file unless there is a specific need. Instead, combine all results into a single file with columns marking scenario and replication IDs.

2. Avoid large file sizes. Be strategic about what you save. For short tests and debugging, saving detailed (“patient-level”) results makes sense. But for full-scale runs with many replications, those files can become unmanageably large. Generally, save summary outputs to file for analysis (e.g., means from each run), not massive raw datasets. If you absolutely need to save or share large files, use compressed formats (e.g., csv.gz). Also, keep in mind practical size limits for version control: for example, GitHub’s individual file size limit is 100 MB.

Explore the example models

Nurse visit simulation

GitHub Click to visit pydesrap_mms repository

Key files simulation/run_scenarios.py
notebooks/analysis.ipynb
What to look for? Similar implementation to above.

GitHub Click to visit rdesrap_mms repository

Key files R/scenarios.R
rmarkdown/analysis.md
What to look for? Similar implementation to above.

Stroke pathway simulation

GitHub Click to visit pydesrap_stroke repository

Key files notebooks/analysis.ipynb
What to look for? These scenarios are run one-by-one in the Jupyter notebook. They involve altering arrivals before running the simulation (5% increase in admissions), or changes in calculations (pooling beds).

GitHub Click to visit rdesrap_stroke repository

Key files rmarkdown/analysis.md
What to look for? These scenarios are run one-by-one in the Rmarkdown. They involve altering arrivals before running the simulation (5% increase in admissions), or changes in calculations (pooling beds).

Test yourself

Try running your own scenario and sensitivity analyses.

You could build your own simple loop to test different values - just make sure you use a fresh set of parameters for each run.

Alternatively, use the run_scenarios function.

References

Heather, Amy, Thomas Monks, Alison Harper, Navonil Mustafee, and Andrew Mayne. 2025. “On the Reproducibility of Discrete-Event Simulation Studies in Health Research: An Empirical Study Using Open Models.” Journal of Simulation 0 (0): 1–25. https://doi.org/10.1080/17477778.2025.2552177.