Improving Reproducibility in Open Healthcare Simulation

---

Amy Heather 0000-0002-6596-3479

Part of the “STARS” project:

Discrete event simulation (DES)

Animation created using web app developed by Sammi Rosser 0000-0002-9552-8988
Available at hsma-programme/Teaching_DES_Concepts_Streamlit (2024) (MIT Licence).

Sharing simulation models

Reproducibility: Generating reported results using the provided code and data.

# Simulation code...
# etc.

# Plotting code...
fig = px.line(
    df, 
    x='Number of Nurses', 
    y='Mean Wait Time (min)', 
    color='Inter-arrival Time',
    markers=True,
    color_discrete_sequence=['#0000FF', '#FFA500', '#FF0000']
)

fig.update_layout(
    title='Impact of Nursing Staff on Patient Wait Times',
    xaxis_title='Number of Nurses',
...

Why does reproducibility matter?

For other modellers…

For modeller…

Trust

Reuse

Reuse

Quality

Reasons for non-reproducility include:

• Missing code
• Lost parameters
• Software changes

Troubleshooting time-consuming and maybe impossible if information is lost.

Research aim:

Generate a set of actionable recommendations to improve reproducibility of healthcare simulation models.

Protocol

Pre-registration

GitHub template

Recommendations

Preprint

Ongoing / future work

Ongoing / future work

Re-runnable

Code runs


 

Repeatable

Code can produce the same results more than once
 

Reproducible

Running code regenerates the published results
 

↓
This work

Reusable

Code can be adapted and used in new contexts
 

↓
STARS ¹

Replicable

New code based on described methods produces consistent results

↓
STRESS ²

• Working with journals “Simulation” and “Journal of Simulation”.
• Developing training materials with HDR UK.

1. Monks, T., Harper, A., & Mustafee, N. (2024). Towards sharing tools and artefacts for reusable simulations in healthcare. Journal of Simulation, 1–20. 10.1080/17477778.2024.2347882

2. Monks, T., Currie, C., Onggo, B., et al. (2019). Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines. Journal of Simulation, 13(1), 55–67. 10.1080/17477778.2018.1442155

Acknowledgements

Thanks for their supervision, support, and involvement to:

• Tom Monks 0000-0003-2631-4481
• Alison Harper 0000-0001-5274-5037
• Nav Mustafee 0000-0002-2204-8924
• Andy Mayne 0000-0003-1263-2286

Thanks to the authors who made their code available under open licences, facilitating this research.

This project is supported by the Medical Research Council [grant number MR/Z503915/1].