Research

Our research has been focused on helping improving the reproducibility, reusability and replicability of DES models. These are defined as follows:

• Reproducible: If all code, data, environment details, and parameters are recorded and published, anyone (including yourself in the future or a different person on a different system) can regenerate the published results by re-creating the documented conditions. This verifies code is working as expected and increases trust.

• Reusable: Code can be adapted and used in new contexts. This saves time and increases impact.

• Replicable: New code based on described methods produces consistent results. This gives confidence in results: their validity and reliability.

Context

Motivation: a failed attempt at reuse

STARS grew out of an unsuccessful attempt to share a stroke simulation model with the NHS in England. That experience showed how important it is for simulation research to be open, reproducible, and reusable by others.

Reproducible

Computational reproducibility assessment of published DES models

We attempted to reproduce eight published Python and R healthcare DES models. From this work, we developed the STARS reproducibility recommendations for sharing reproducible healthcare DES models.

DES RAP Book and four worked examples

This comprehensive e-book provides step-by-step instructions for building DES models in Python or R as part of a reproducible analytical pipeline (RAP). It is accompanied by four full example repositories.

Journal of Simulation - Model Reproducibility Initiative

This new initiative lets authors submit their simulation models for review, awarding badges that show the word is open, reviewed, reproduced and/or reusable.

HDR UK Futures training materials

In development - watch this space!

Reusable

DES sharing review

This review assessed the extent to which healthcare DES models are shared, and audited whether sharing adhered to best practices.

STARS framework for model reuse

The STARS reuse framework is a set of practical guidelines, with essential and optional components, that supports healthcare DES modellers to share open-source models in a way that makes them easier to reuse.

Exeter Oncology Model: Renal Cell Carcinoma edition

We worked with an oncology cost‑effectiveness model, developed by PenTAG and collaborators for the NICE Pathways Pilot appraisal, to align it with the STARS framework for reusable simulations. Our contribution largely focused on developing clear, user‑oriented documentation and a pilot web application.

Reproducible and reusable

Journal of Simulation - Model Reproducibility Initiative

This new initiative lets authors submit their simulation models for review, awarding badges that show the word is open, reviewed, reproduced and/or reusable.

Reviewing models to support authors in making their work reproducible and reusable

We worked with model authors to review their DES models, providing guidance to help improve reproducibility and reusability.

Replicable

STRESS

This was not developed as part of the STARS project, but is related work that Tom was involved in. The STRESS reporting guidelines are designed to improve the replicability of simulation models.

Other avenues of work

Using LLMs to generate DES models

This study investigated whether generative AI could replicate published healthcare DES models.

sim-tools

Our python package with tools to support DES and Monte-Carlo simulation.