Reporting guidelines

This page evaluates the extent to which the journal article meets the criteria from two discrete-event simulation study reporting guidelines:

STRESS-DES

Of the 24 items in the checklist:

  • 22 were met fully (✅)
  • 2 were partially met (🟡)

Although this is unsurprising, as they state that they used STRESS-DES when reporting and - in the paper that first describes the model that this paper built upon - they even attach the completed checklist.

Item Recommendation Met by study? Evidence
Objectives
1.1 Purpose of the model Explain the background and objectives for the model ✅ Fully Detailed in introduction/background e.g. “purpose of this study is to investigate the likely impact of various strategies for triaging admission to intensive care during the COVID-19 pandemic”

Wood et al. (2021)
1.2 Model outputs Define all quantitative performance measures that are reported, using equations where necessary. Specify how and when they are calculated during the model run along with how any measures of error such as confidence intervals are calculated. ✅ Fully In introduction/background - “The primary outcome measures are aggregate lives and life-years saved relative to the baseline involving no triage (in which patients are admitted on a first-come, first-served basis)”.”

Supplementary materials - “Outputs are then aggregated across these replications, with central estimates (based on the mean) and confidence intervals (at the 95% level) calculated for all considered performance measures.”

Wood et al. (2021)
1.3 Experimentation aims If the model has been used for experimentation, state the objectives that it was used to investigate.
(A) Scenario based analysis – Provide a name and description for each scenario, providing a rationale for the choice of scenarios and ensure that item 2.3 (below) is completed.
(B) Design of experiments – Provide details of the overall design of the experiments with reference to performance measures and their parameters (provide further details in data below).
(C) Simulation Optimisation – (if appropriate) Provide full details of what is to be optimised, the parameters that were included and the algorithm(s) that was be used. Where possible provide a citation of the algorithm(s).		 ✅ Fully The scenarios are clearly visualised in Figure 1 and described in Methods: Triage Strategies - e.g. “The first strategy accounts for a rigid cutoff, in which no patient is admit- ted to intensive care whose age is above the considered threshold (cutoff strategy). The second strategy relaxes this constraint, to the extent that…”. Then in Methods: Triage Strategies, “activities were simulated for a 20-bed intensive care unit… To gauge the sensitivity to different ward sizes, modeling was also performed on ward sizes ranging from 10 to 200 beds”. Then in Methods: Application, e.g. “3 demand trajectories for intensive care admission were synthetically generated with the aim of stressing the bed base…”

Wood et al. (2021)
Logic
2.1 Base model overview diagram Describe the base model using appropriate diagrams and description. This could include one or more process flow, activity cycle or equivalent diagrams sufficient to describe the model to readers. Avoid complicated diagrams in the main text. The goal is to describe the breadth and depth of the model with respect to the system being studied. ✅ Fully Figure 2
2.2 Base model logic Give details of the base model logic. Give additional model logic details sufficient to communicate to the reader how the model works. ✅ Fully Figure 2 and Methods: Model
2.3 Scenario logic Give details of the logical difference between the base case model and scenarios (if any). This could be incorporated as text or where differences are substantial could be incorporated in the same manner as 2.2. ✅ Fully Figure 1 and Methods: Triage Strategies
2.4 Algorithms Provide further detail on any algorithms in the model that (for example) mimic complex or manual processes in the real world (i.e. scheduling of arrivals/ appointments/ operations/ maintenance, operation of a conveyor system, machine breakdowns, etc.). Sufficient detail should be included (or referred to in other published work) for the algorithms to be reproducible. Pseudo-code may be used to describe an algorithm. ✅ Fully Methods: Application - e.g. “Demand for intensive care admission was generated using a Susceptible-Exposed-Infected-Recovered compartmental model41 developed for use within the COVID-19 setting…” - and Table 2

Wood et al. (2021)
2.5.1 Components - entities Give details of all entities within the simulation including a description of their role in the model and a description of all their attributes. ✅ Fully Patients with an age - as mentioned in Methods: Application, “Patient groups are defined by the 6 age brackets provided in the weekly reports published by the Intensive Care National Audit and Research Centre (ICNARC)” and given in Table 1

Wood et al. (2021)
2.5.2 Components - activities Describe the activities that entities engage in within the model. Provide details of entity routing into and out of the activity. ✅ Fully Evident in Figure 2 - admission (which can be accepted, declined, or interrupted)
2.5.3 Components - resources List all the resources included within the model and which activities make use of them. ✅ Fully Implicit in Methods: Triage Strategies, with activity being admission, and resource being bed - “patients are admit- ted provided there is at least a certain number of beds”

Wood et al. (2021)
2.5.4 Components - queues Give details of the assumed queuing discipline used in the model (e.g. First in First Out, Last in First Out, prioritisation, etc.). Where one or more queues have a different discipline from the rest, provide a list of queues, indicating the queuing discipline used for each. If reneging, balking or jockeying occur, etc., provide details of the rules. Detail any delays or capacity constraints on the queues. ✅ Fully As in Methods, “the modeled queue discipline from the originally assumed ‘first-come, first-served’ to allow for the 3 triage strategies outlined in the ‘Triage Strategies’ section.”… and these are then described in Methods: Triage Strategies.

Wood et al. (2021)
2.5.5 Components - entry/exit points Give details of the model boundaries i.e. all arrival and exit points of entities. Detail the arrival mechanism (e.g. ‘thinning’ to mimic a non-homogenous Poisson process or balking) ✅ Fully Evident in Figure 1 and Figure 2.
Data
3.1 Data sources List and detail all data sources. Sources may include:
• Interviews with stakeholders,
• Samples of routinely collected data,
• Prospectively collected samples for the purpose of the simulation study,
• Public domain data published in either academic or organisational literature. Provide, where possible, the link and DOI to the data or reference to published literature.
All data source descriptions should include details of the sample size, sample date ranges and use within the study.		 ✅ Fully Described in Methods, Table 3 and Supplementary Materials - e.g. “Patient groups are defined by the 6 age brackets provided in the weekly reports published by the Intensive Care National Audit and Research Centre (ICNARC; https://www.icnarc.org)”… “number of life-years remaining was calculated as the sex-weighted mean value for each age group using UK national life tables pub- lished by the Office for National Statistics”

Wood et al. (2021)
3.2 Pre-processing Provide details of any data manipulation that has taken place before its use in the simulation, e.g. interpolation to account for missing data or the removal of outliers. ✅ Fully Described in Methods: Application - e.g. generating demand trajectories using the Susceptible-Exposed-Infected-Recovered model as described in Figure 3, and e.g. description ofhow data from different sources is used, like the deduced median and interquartile range used to approprixmate distributions of lengths of stay.

Further supported by descriptions in the STRESS-DES checklist in the Supplementary Materials of Wood et al. (2020) (paper with model that Wood et al. 2021 built upon).
3.3 Input parameters List all input variables in the model. Provide a description of their use and include parameter values. For stochastic inputs provide details of any continuous, discrete or empirical distributions used along with all associated parameters. Give details of all time dependent parameters and correlation.
Clearly state:
• Base case data
• Data use in experimentation, where different from the base case.
• Where optimisation or design of experiments has been used, state the range of values that parameters can take.
• Where theoretical distributions are used, state how these were selected and prioritised above other candidate distributions.		 ✅ Fully Table 1, Table 2, Table 3, and other/supporting descriptions in Methods and Supplementary Materials.
3.4 Assumptions Where data or knowledge of the real system is unavailable what assumptions are included in the model? This might include parameter values, distributions or routing logic within the model. ✅ Fully Mentioned throughout paper - e.g. “The probability that a patient will die if not admitted to inten- sive care was assumed uniform across all patient groups…”, “It was assumed—in absence of available data— that patients discharged prematurely, at any number of days postadmission, had the same probability of death…”, “A key assumption of the interrupt strategy has been that a patient may be prematurely discharged at any number of days after admission…”

Wood et al. (2021)
Experimentation
4.1 Initialisation Report if the system modelled is terminating or non-terminating. State if a warm-up period has been used, its length and the analysis method used to select it. For terminating systems state the stopping condition.
State what if any initial model conditions have been included, e.g., pre-loaded queues and activities. Report whether initialisation of these variables is deterministic or stochastic.		 ✅ Fully Couldn’t spot mention in this paper, but it does state in the paper and supplementary materials that this is based on a previous model (and then describes the specific modifications made from that model). As such, in line with previous evaluations (where we have assumed the information can be provided in an earlier paper, as long as they would not have changed in the current paper), I checked the prior paper for this detail, and it does state: “events are iterated in line with the three-phased method until some terminating criterion is met. Here, this is given by the time at which some outcome has been reached for all simulated admissions for the given epidemic curve (for cases requiring intensive care admission), i.e. each sought admission has been either rejected or admitted and discharged or died (event types c-f)”

Wood et al. (2020)
4.2 Run length Detail the run length of the simulation model and time units. ✅ Fully As stated in the STRESS-DES checklist in the Supplementary Materials of Wood et al. (2020) (paper with model that Wood et al. 2021 built upon), “Determined by the stopping condition outlined in 4.1 of this checklist.”
4.3 Estimation approach State the method used to account for the stochasticity: For example, two common methods are multiple replications or batch means. Where multiple replications have been used, state the number of replications and for batch means, indicate the batch length and whether the batch means procedure is standard, spaced or overlapping. For both procedures provide a justification for the methods used and the number of replications/size of batches. ✅ Fully Supplementary materials - 1000 replications. This is justified in prior paper in 2.4 Simulation: “This number of replications was selected based on the resulting reduction of simulation error to magnitudes deemed sufficiently negligible (<0.25%) when assessed against the output measures of interest (this was performed using different seeds for which the random number streams were drawn for each replication within the simulations considered).”

Wood et al. (2020)
Implementation
5.1 Software or programming language State the operating system and version and build number.
State the name, version and build number of commercial or open source DES software that the model is implemented in.
State the name and version of general-purpose programming languages used (e.g. Python 3.5).
Where frameworks and libraries have been used provide all details including version numbers.		 🟡 Partially Supplementary materials - “All simulations were performed on a Windows 10 desktop computer using 64-bit R version 3.6.0.

Doesn’t given libraries and their versions.

Wood et al. (2021)
5.2 Random sampling State the algorithm used to generate random samples in the software/programming language used e.g. Mersenne Twister.
If common random numbers are used, state how seeds (or random number streams) are distributed among sampling processes.		 ✅ Fully Supplementary Materials: “In order to appreciate the many other possibilities a number of ‘replications’ are performed, each drawing upon different random number streams for sampling the various probabilities and lengths of stay.”

Wood et al. (2021)

Further supported by descriptions in the STRESS-DES checklist in the Supplementary Materials of Wood et al. (2020) - “Uses the inbuilt random number generator in R. Each replication uses a different seed call to this function.”
5.3 Model execution State the event processing mechanism used e.g. three phase, event, activity, process interaction.
Note that in some commercial software the event processing mechanism may not be published. In these cases authors should adhere to item 5.1 software recommendations.
State all priority rules included if entities/activities compete for resources.
If the model is parallel, distributed and/or use grid or cloud computing, etc., state and preferably reference the technology used. For parallel and distributed simulations the time management algorithms used. If the HLA is used then state the version of the standard, which run-time infrastructure (and version), and any supporting documents (FOMs, etc.)		 ✅ Fully Three phase, described in detail in 2.1 Model and STRESS-DES checklist in supplementary materials of Wood et al. (2020)
5.4 System specification State the model run time and specification of hardware used. This is particularly important for large scale models that require substantial computing power. For parallel, distributed and/or use grid or cloud computing, etc. state the details of all systems used in the implementation (processors, network, etc.) 🟡 Partially Stated run time in this paper and prior, and explains why hardware is not relevant in prior - “Processing time is insubstantial, typically taking less than five minutes for each scenario evaluated on a desktop computer (note that scenarios with larger projections of number of admissions than those considered here take longer due to more “discrete events” taking place). Computational constraints are on processing time and not computer memory.” (Wood et al. (2020)). However, given that I found the model took 48 hours to run, I feel this description would benefit from mentioning that, and feels incomplete.
Code access
6.1 Computer model sharing statement Describe how someone could obtain the model described in the paper, the simulation software and any other associated software (or hardware) needed to reproduce the results. Provide, where possible, the link and DOIs to these. ✅ Fully Research Data - “The data sets generated and/or analyzed during the current study are available in the ‘triage-modelling’ repository, https://github.com/nhs-bnssg-analytics/triage-modelling.”

Wood et al. (2021)

DES checklist derived from ISPOR-SDM

Of the 18 items in the checklist:

  • 13 were met fully (✅)
  • 4 were not met (❌)
  • 1 was not applicable (N/A)
Item Assessed if… Met by study? Evidence/location
Model conceptualisation
1 Is the focused health-related decision problem clarified? …the decision problem under investigation was defined. DES studies included different types of decision problems, eg, those listed in previously developed taxonomies. ✅ Fully Introduction/background, e.g. “COVID-19 is a virulent disease caused by the highly con- tagious severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). With an R0, or basic reproduction num- ber, estimated as high as 6.5, it has proliferated globally… Regarding the particularly severe pressure put on intensive care resources during a pandemic, the research interest in triage strategy intensified…”

Wood et al. (2021)
2 Is the modeled healthcare setting/health condition clarified? …the physical context/scope (eg, a certain healthcare unit or a broader system) or disease spectrum simulated was described. ✅ Fully Introduction/background - e.g. “estimated 15% of symptomatic UK cases requir- ing hospital admission…. significant proportion (17%) of hospitalized patients require transfer to intensive care,2 of which 41% eventually die within intensive care…”

Wood et al. (2021)
3 Is the model structure described? …the model’s conceptual structure was described in the form of either graphical or text presentation. ✅ Fully Figure 2 and Methods: Model
4 Is the time horizon given? …the time period covered by the simulation was reported. ✅ Fully As stated in the STRESS-DES checklist in the Supplementary Materials of Wood et al. (2020) (paper with model that Wood et al. 2021 built upon), “Determined by the stopping condition outlined in 4.1 of this checklist.”
5 Are all simulated strategies/scenarios specified? …the comparators under test were described in terms of their components, corresponding variations, etc ✅ Fully The scenarios are clearly visualised in Figure 1 and described in Methods: Triage Strategies - e.g. “The first strategy accounts for a rigid cutoff, in which no patient is admit- ted to intensive care whose age is above the considered threshold (cutoff strategy). The second strategy relaxes this constraint, to the extent that…”. Then in Methods: Triage Strategies, “activities were simulated for a 20-bed intensive care unit… To gauge the sensitivity to different ward sizes, modeling was also performed on ward sizes ranging from 10 to 200 beds”. Then in Methods: Application, e.g. “3 demand trajectories for intensive care admission were synthetically generated with the aim of stressing the bed base…”

Wood et al. (2021)
6 Is the target population described? …the entities simulated and their main attributes were characterized. ✅ Fully Table 1
Paramaterisation and uncertainty assessment
7 Are data sources informing parameter estimations provided? …the sources of all data used to inform model inputs were reported. ✅ Fully Described in Methods, Table 3 and Supplementary Materials - e.g. “Patient groups are defined by the 6 age brackets provided in the weekly reports published by the Intensive Care National Audit and Research Centre (ICNARC; https://www.icnarc.org)”… “number of life-years remaining was calculated as the sex-weighted mean value for each age group using UK national life tables pub- lished by the Office for National Statistics”

Wood et al. (2021)
8 Are the parameters used to populate model frameworks specified? …all relevant parameters fed into model frameworks were disclosed. ✅ Fully Table 1, Table 2, Table 3, and other/supporting descriptions in Methods and Supplementary Materials.
9 Are model uncertainties discussed? …the uncertainty surrounding parameter estimations and adopted statistical methods (eg, 95% confidence intervals or possibility distributions) were reported. ✅ Fully Reports mean and confidence intervals in Table 4.
10 Are sensitivity analyses performed and reported? …the robustness of model outputs to input uncertainties was examined, for example via deterministic (based on parameters’ plausible ranges) or probabilistic (based on a priori-defined probability distributions) sensitivity analyses, or both. ✅ Fully Perform sensitivity analysis varying bed capacity from 10 to 200, with results in Figure 7.
Validation
11 Is face validity evaluated and reported? …it was reported that the model was subjected to the examination on how well model designs correspond to the reality and intuitions. It was assumed that this type of validation should be conducted by external evaluators with no stake in the study. ❌ Not met -
12 Is cross validation performed and reported …comparison across similar modeling studies which deal with the same decision problem was undertaken. ❌ Not met -
13 Is external validation performed and reported? …the modeler(s) examined how well the model’s results match the empirical data of an actual event modeled. ❌ Not met -
14 Is predictive validation performed or attempted? …the modeler(s) examined the consistency of a model’s predictions of a future event and the actual outcomes in the future. If this was not undertaken, it was assessed whether the reasons were discussed. N/A This is only relevant to forecasting models.
Generalisability and stakeholder involvement
15 Is the model generalizability issue discussed? …the modeler(s) discussed the potential of the resulting model for being applicable to other settings/populations (single/multiple application). ✅ Fully Discussion: Limitations and Further Research: “First, on the grounds of geography, it is important to note that the data-driven aspects of model parameterization were derived from the UK expe- rience (Table 3). Investigators seeking to meaningfully transfer the results of this study to another geography Medical Decision Making 41(4) should consider the extent to which assumptions are aligned (e.g., with respect to lengths of stay). Where sub- stantial differences remain, adapting the open-source model code to perform a set of modified simulation experiments may be a consideration.”

Wood et al. (2021)
16 Are decision makers or other stakeholders involved in modeling? …the modeler(s) reported in which part throughout the modeling process decision makers and other stakeholders (eg, subject experts) were engaged. ❌ Not met No mention of this.
17 Is the source of funding stated? …the sponsorship of the study was indicated. ✅ Fully Funding Information: “The authors received no financial support for the research, authorship, and/or publication of this article.”
18 Are model limitations discussed? …limitations of the assessed model, especially limitations of interest to decision makers, were discussed. ✅ Fully Discussion: Limitations and Further Research

References

Monks, Thomas, Christine S. M. Currie, Bhakti Stephan Onggo, Stewart Robinson, Martin Kunc, and Simon J. E. Taylor. 2019. “Strengthening the Reporting of Empirical Simulation Studies: Introducing the STRESS Guidelines.” Journal of Simulation 13 (1): 55–67. https://doi.org/10.1080/17477778.2018.1442155.
Wood, Richard M., Christopher J. McWilliams, Matthew J. Thomas, Christopher P. Bourdeaux, and Christos Vasilakis. 2020. “COVID-19 Scenario Modelling for the Mitigation of Capacity-Dependent Deaths in Intensive Care.” Health Care Management Science 23 (3): 315–24. https://doi.org/10.1007/s10729-020-09511-7.
Wood, Richard M., Adrian C. Pratt, Charlie Kenward, Christopher J. McWilliams, Ross D. Booton, Matthew J. Thomas, Christopher P. Bourdeaux, and Christos Vasilakis. 2021. “The Value of Triage During Periods of Intense COVID-19 Demand: Simulation Modeling Study.” Medical Decision Making 41 (4): 393–407. https://doi.org/10.1177/0272989X21994035.
Zhang, Xiange, Stefan K. Lhachimi, and Wolf H. Rogowski. 2020. “Reporting Quality of Discrete Event Simulations in Healthcare—Results From a Generic Reporting Checklist.” Value in Health 23 (4): 506–14. https://doi.org/10.1016/j.jval.2020.01.005.