Day 3

reproduction
report
evaluation
compendium
Author

Amy Heather

Published

August 13, 2024

Note

Finished reproduction of the single figure. Total time used: 2h 10m (5.4%). Evaluated against criteria and guidelines. Evaluation time: 1h 12m. Then completed summary report, reflections, and research compendium stage.

09.14-09.38: Resuming recreation of figure

Continuing from yesterday. I included the running of the model within reproduction.ipynb, and finished up the Figure so it matches the paper.

Feel this is reproduced at 09.38.

Untimed: Reproduction success

Filled in reproduction success page

Timings for reproduction

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 106

# Times from today
times = [
    ('09.14', '09.38')]

calculate_times(used_to_date, times)
Time spent today: 24m, or 0h 24m
Total used to date: 130m, or 2h 10m
Time remaining: 2270m, or 37h 50m
Used 5.4% of 40 hours max

09.47-10.00: STARS evaluation

Evaluate code against STARS framework.

10.03-10.09: Badges

Evaluate code against against journal badge criteria. Explaining some of the decisions:

  • Complete set of materials - no ❌, as didn’t include code to create the figure itself
  • Carefully documented - yes ✅, considering the support of CHARM_INFO.md alongside the README.md to explain what the inputs were (hence support reuse and change of those inputs, since it explains that we run by calling .csv files with inputs)
  • README with step-by-step instructions - yes ✅, I was a little more uncertain on this one, since the README doesn’t explicitly say how to make the figure, but it does provide instructions that lead you to regenerate the exact model results from the paper, and so I feel that it does provide instructions to reproduce results sufficiently (although would be more complete to include instructions for figure too - so if it weren’t a yes/no decision for badges, I would’ve said this was partially met).

10.17-10.46: STRESS-DES

Evaluate paper against STRESS-DES reporting guidelines.

10.59-11.23: ISPOR-SDM

Evaluate paper against guidelines from Zhang, Lhachimi, and Rogowski (2020) derived from ISPOR-SDM. Explaining some of the decisions:

  • Run length - not provided. Can understand that this is because this is an example paper, but as it does not provide an example run length, or range of typical/expected/good run lengths, I feel this is not met (as that could have been provided).

Timings for evaluation

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 0

# Times from today
times = [
    ('09.47', '10.00'),
    ('10.03', '10.09'),
    ('10.17', '10.46'),
    ('10.59', '11.23')]

calculate_times(used_to_date, times, limit=False)
Time spent today: 72m, or 1h 12m
Total used to date: 72m, or 1h 12m

Untimed: Reflections

Wrote up reflections.qmd based on logbook entries.

Untimed: Summary report

Completed summary report using template.

Untimed: Research compendium

Seperate folders for data, methods and outputs: Moved scripts into a script folder. Had to modify utils.py as:

  • That automatically adds ./input to your input file paths, but we need ../input.
  • Path to output file is different now too

Tests:

  • Add pytest to environment
  • Modify utils.py as it assumes you always want to save the results as a csv file in ../output/OUT_STATS.csv. Tried to amend this following the structure they had already set up for the input file paths. Note: It was quite tricky to do this - this was pretty hard coded into the script.

Run time: Add note that run time is only a couple of seconds to the notebook and README

Dockerfile: Created Dockerfile as I did for Shoaib et al. 2022.

  • Add conda-forge channel, plus jupyterlab and notebook packages, to environment
  • Built image, created container, opened JupyterLab, and ran both the notebook and test successfully.

GitHub container repository:

  • Activated GitHub action
  • Pulled container and tested running notebook and test successfully

README: Completed the README.

References

Zhang, Xiange, Stefan K. Lhachimi, and Wolf H. Rogowski. 2020. “Reporting Quality of Discrete Event Simulations in Healthcare—Results From a Generic Reporting Checklist.” Value in Health 23 (4): 506–14. https://doi.org/10.1016/j.jval.2020.01.005.