Reproducing Kim et al. 2021

Note

Finished reproduction (supplementary table 2 and figure). Completed evaluation against guidelines. Started on research compendium. Total reproduction time: 14h 35m (36.5%). Evaluation time: 1h 55m.

09.09-09.43: Resuming supplementary table 2

Combined each scenario with scenario 0, then adapted previous functions to generate the table.

Consider this reproduced at 09.43.

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 805

# Times from today
times = [
    ('09.09', '09.43')]

calculate_times(used_to_date, times)

Time spent today: 34m, or 0h 34m
Total used to date: 839m, or 13h 59m
Time remaining: 1561m, or 26h 1m
Used 35.0% of 40 hours max

10.02-10.38: Supplementary figure 3

Amend prior functions to combine with scenario 0 using a function (as repeated lots, and need again for this figure). Could then use those dataframes in my figure functions.

Consider this reproduced at 10.38.

Reproduction timings

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 805

# Times from today
times = [
    ('09.09', '09.43'),
    ('10.02', '10.38')]

calculate_times(used_to_date, times)

Time spent today: 70m, or 1h 10m
Total used to date: 875m, or 14h 35m
Time remaining: 1525m, or 25h 25m
Used 36.5% of 40 hours max

10.54-10.59: Badges

Evaluated repository as available at this time (so including the few commits since publication).

documentation_sufficient - no as no package versions and as most instructions are within R scripts themselves (e.g. that need to run with source()) (and therefore not met the subsequent careful documentation)

documentation_readme - although it directs to a good example file, it doesn’t include instructions for reproducing results from the article

11.00-11.05, 11.18-11.23, 11.29-11.32: STARS framework

11.33-12.10, 12.55-13.10, 13.15-13.30: STRESS-DES

To discuss (combined with ISPOR-SDM when overlap):

2.1 Base model overview diagram / 3 Is the model structure described? - diagram is in prior study
2.2 Base model logic - described in detail in prior study, only briefly alluded to in this one
2.4 Algorithms - partially described in prior study
2.5.2 Components - activities - provided moreso in prior study
2.5.3 Components - resources - not clear in either study, unlesss there are none?
2.5.4 Components - queues - not clear in either study, unless there are none?
4.1 Initialisation - indicates its non-terminating in prior study
5.3 Model execution - prior study mentions that you can run in parallel
5.4 System specification - prior study mentions HPC

13.48-14.18: ISPOR-SDM

Evaluation timings

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 0

# Times from today
times = [
    ('10.54', '10.59'),
    ('11.00', '11.05'),
    ('11.18', '11.23'),
    ('11.29', '11.32'),
    ('11.33', '12.10'),
    ('12.55', '13.10'),
    ('13.15', '13.30'),
    ('13.48', '14.18')]

calculate_times(used_to_date, times, limit=False)

Time spent today: 115m, or 1h 55m
Total used to date: 115m, or 1h 55m

Untimed: Summary report

Partially completed summary report (evaluation will need to be updated following second opinion, as there were several items I marked as TBC, as I wasn’t certain whether they met criteria when the information was from a prior study).

Untimed: Reflections

Detailed troubleshooting steps and add reflections.