STARS framework
This page evaluates the extent to which the original study meets the recommendations from the STARS framework for the sharing of code and associated materials from discrete-event simulation models (Monks, Harper, and Mustafee (2024)).
Of the 8 essential STARS components:
- 4 were met fully (✅)
- 2 were met partially (🟡)
- 2 were not met (❌)
Of the 5 optional STARS components:
- 5 were not met (❌)
| Component | Description | Met by study? | Evidence/location |
|---|---|---|---|
| Essential components | |||
| Open license | Free and open-source software (FOSS) license (e.g. MIT, GNU Public License (GPL)) | ✅ Fully | GNU-3 |
| Dependency management | Specify software libraries, version numbers and sources (e.g. dependency management tools like virtualenv, conda, poetry) | 🟡 Partially | Rationale for partial rating: • Instructions in GitHub were to install this directly from GitHub (which would install dependencies too) but that is not possible as it depends on rredis as an Import but that has been archived and so requires remotes to install • Some of the dependencies used in the .Rmd files for the analysis were not in DESCRIPTION (as well as some dependencies that appeared as missing import errors)• Doesn’t specify all versions (which, although not necessary to run, would be helpful as a “snapshot” of what the analysis ran with for the paper). |
| FOSS model | Coded in FOSS language (e.g. R, Julia, Python) | ✅ Fully | R |
| Minimum documentation | Minimal instructions (e.g. in README) that overview (a) what model does, (b) how to install and run model to obtain results, and (c) how to vary parameters to run new experiments | 🟡 Partially | README has mini description of model, and clear instructions on how to install and run model, but doesn’t explain how to vary parameters, and so have marked as partially met. |
| ORCID | ORCID for each study author | ❌ Not met | - |
| Citation information | Instructions on how to cite the research artefact (e.g. CITATION.cff file) | ✅ Fully | Includes instructions for citation. |
| Remote code repository | Code available in a remote code repository (e.g. GitHub, GitLab, BitBucket) | ✅ Fully | https://github.com/KateJohnson/epicR/tree/closed_cohort |
| Open science archive | Code stored in an open science archive with FORCE11 compliant citation and guaranteed persistance of digital artefacts (e.g. Figshare, Zenodo, the Open Science Framework (OSF), and the Computational Modeling in the Social and Ecological Sciences Network (CoMSES Net)) | ❌ Not met | As far as I am aware. |
| Optional components | |||
| Enhanced documentation | Open and high quality documentation on how the model is implemented and works (e.g. via notebooks and markdown files, brought together using software like Quarto and Jupyter Book). Suggested content includes: • Plain english summary of project and model • Clarifying license • Citation instructions • Contribution instructions • Model installation instructions • Structured code walk through of model • Documentation of modelling cycle using TRACE • Annotated simulation reporting guidelines • Clear description of model validation including its intended purpose |
❌ Not met | - |
| Documentation hosting | Host documentation (e.g. with GitHub pages, GitLab pages, BitBucket Cloud, Quarto Pub) | ❌ Not met | - |
| Online coding environment | Provide an online environment where users can run and change code (e.g. BinderHub, Google Colaboratory, Deepnote) | ❌ Not met | The README mentions that the model is on PRISM but from the links, one doesn’t work and one links to a page that simply has a description of the model and copy of the README. Referring to the original EPIC repository it is forked from, there is a working link to the Peer Models Network, which contains several links and instructions on how to install epic from GitHub. This is great for helping bring together collections of models with relevant instructions and links but, from looking at it, is not an online coding environment (and is more relevant to the full EPIC repository than that for this study regardless), so marked as not met. |
| Model interface | Provide web application interface to the model so it is accessible to less technical simulation users | ❌ Not met | - |
| Web app hosting | Host web app online (e.g. Streamlit Community Cloud, ShinyApps hosting) | ❌ Not met | - |
References
Monks, Thomas, Alison Harper, and Navonil Mustafee. 2024. “Towards Sharing Tools and Artefacts for Reusable Simulations in Healthcare.” Journal of Simulation 0 (0): 1–20. https://doi.org/10.1080/17477778.2024.2347882.