Sharing research artefacts
This page evaluates the extent to which the original study meets the recommendations for the sharing of code and associated materials. We compare against two standards:
- Best practice audit - requirements from the best practice audit by Monks and Harper (2023). The items in this audit were based on The Turing Way Community (2022), Taylor et al. (2017), and The Open Modeling Foundation (OMF) (2024).
- STARS - recommendations from the recently published STARS framework by Monks, Harper, and Mustafee (2024).
Best practice audit
Note: This was removed from the protocol after this test-run due to overlap with the STARS framework, which was designed to build on these criteria
Of the 11 items in this audit:
- 9 were met fully (✅)
- 1 was met partially (🟡)
- 1 was not met (❌)
| Item | Description | Met by study? | Evidence/location |
|---|---|---|---|
| Digital Object Identifier (DOI) | Does the model have a DOI and promise of persistence? Can it be cited? | ✅ Fully | 10.5281/zenodo.3760625 |
| Open Researcher and Contributor ID (ORCID) | Is the model linked to one or more of the authors via an ORCID? | ✅ Fully | Monks linked to ORCID on Zenodo and in journal article |
| License | Does the repository have a recognised open license to control the use of code, liabilty and credit? | ✅ Fully | MIT license |
| Model overview | Is there an obvious file that provides an overview of the repository/model and it purpose (e.g. in README file)? | ✅ Fully | README.md present but doesn’t overview repository/model or purpose, although does like to Capacity_Model_Template.ipynb which does overview the model |
| Link to published paper | Do models shared externally from journal articles contain a link to the published article? | 🟡 Partially | Zenodo citations section includes link to the article |
| Steps to run code | Does the readme file or similar describe the steps required to execute the simulation model? | ✅ Fully | Provided in Capacity_Model_Template.ipynb |
| Formal dependency management | Has a formal tool, e.g. renv, conda, or poetry been used to manage software dependencies for the simulation model? | ✅ Fully | Conda - environment.yaml |
| Informal dependency management | Has an informal list or description of software, or OS dependencies been provided? | ✅ Fully | Within environment.yaml |
| Code testing | Is there any evidence of tests that have been applied to the code to check that it functions correctly? | ❌ Not met | No tests found for DES model (tests for Monte Carlo model are available in GitLab repository) |
| Local execution | Can the simulation model and associated files be downloaded and in theory executed on a local machine | ✅ Fully | I have successfully executed |
| Remote execution | Can the simulation model be executed online using free or commercial infrastructure? | ✅ Fully | The archived code on Zenodo has not been set up to allow this but the production code on GitLab was set up to allow this (on 23rd April 2020, prior to date journal received article) |
STARS
Of the 8 essential STARS components:
- 5 were met fully (✅)
- 2 were met partially (🟡)
- 1 was not met (❌)
Of the 5 optional STARS components:
- 1 was met fully (✅)
- 4 were not met (❌)
| Component | Description | Met by study? | Evidence/location |
|---|---|---|---|
| Essential components | |||
| Open license | Free and open-source software (FOSS) license (e.g. MIT, GNU Public License (GPL)) | ✅ Fully | MIT license |
| Dependency management | Specify software libraries, version numbers and sources (e.g. dependency management tools like pip virtual environment, conda environment, poetry) | ✅ Fully | Conda - environment.yaml |
| FOSS model | Coded in FOSS language (e.g. R, Julia, Python) | ✅ Fully | Python |
| Minimum documentation | Minimal instructions (e.g. in README) that overview (a) what model does, (b) how to install and run model to obtain results, and (c) how to vary parameters to run new experiements | 🟡 Partially | Has README.md which gives a very short description of the model and some instructions on installation, but with little detail, and with no information on varying parameters |
| ORCID | ORCID for each study author | 🟡 Partially | Only provided for Monks |
| Citation information | Instructions on how to cite the research artefact (e.g. CITATION.cff file) | ❌ Not met | Not provided |
| Remote code repository | Code available in a remote code repository (e.g. GitHub, GitLab, BitBucket) | ✅ Fully | GitLab |
| Open science archive | Code stored in an open science archive with FORCE11 compliant citation and guaranteed persistance of digital artefacts (e.g. Figshare, Zenodo, the Open Science Framework (OSF), and the Computational Modeling in the Social and Ecological Sciences Network (CoMSES Net)) | ✅ Fully | Zenodo |
| Optional components | |||
| Enhanced documentation | Open and high quality documentation on how the model is implemented and works (e.g. via notebooks and markdown files, brought together using software like Quarto and Jupyter Book). Suggested content includes: • Plain english summary of project and model • Clarifying license • Citation instructions • Contribution instructions • Model installation instructions • Structured code walk through of model • Documentation of modelling cycle using TRACE • Annotated simulation reporting guidelines • Clear description of model validation including its intended purpose |
❌ Not met | README.md has simple installation instructions, whilst Capacity_Model_Template.ipynb has plain english summary of model and simple walk through. However, many of these suggestions are not captured, and documentation is felt to just meet the criteria of “minimum documentation” |
| Documentation hosting | Host documentation (e.g. with GitHub pages, GitLab pages, BitBucket Cloud, Quarto Pub) | ❌ Not met | Not hosted. |
| Online coding environment | Provide an online environment where users can run and change code (e.g. BinderHub, Google Colaboratory, Deepnote) | ✅ Fully | The archived code on Zenodo has not been set up to allow this but the production code on GitLab was set up to allow this (on 23rd April 2020, prior to date journal received article) |
| Model interface | Provide web application interface to the model so it is accessible to less technical simulation users | ❌ Not met | - |
| Web app hosting | Host web app online (e.g. Streamlit Community Cloud, ShinyApps hosting) | ❌ Not met | - |
References
Monks, Thomas, and Alison Harper. 2023. “Computer Model and Code Sharing Practices in Healthcare Discrete-Event Simulation: A Systematic Scoping Review.” Journal of Simulation 0 (0): 1–16. https://doi.org/10.1080/17477778.2023.2260772.
Monks, Thomas, Alison Harper, and Navonil Mustafee. 2024. “Towards Sharing Tools and Artefacts for Reusable Simulations in Healthcare.” Journal of Simulation 0 (0): 1–20. https://doi.org/10.1080/17477778.2024.2347882.
Taylor, Simon J. E., Anastasia Anagnostou, Adedeji Fabiyi, Christine Currie, Thomas Monks, Roberto Barbera, and Bruce Becker. 2017. “Open Science: Approaches and Benefits for Modeling & Simulation.” In 2017 Winter Simulation Conference (WSC), 535–49. https://doi.org/10.1109/WSC.2017.8247813.
The Open Modeling Foundation (OMF). 2024. “Reusability Standards.” {OMF}. https://www.openmodelingfoundation.org/standards/reusability/.
The Turing Way Community. 2022. “The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research (1.0.2).” Zenodo. https://doi.org/10.5281/zenodo.7625728.