Reporting guidelines

This page evaluates the extent to which the journal article meets the criteria from two discrete-event simulation study reporting guidelines:

Note: This reproduction just focuses on the “CHARM” model from this paper.

STRESS-DES

Of the 24 items in the checklist:

  • 14 were met fully (✅)
  • 2 were partially met (🟡)
  • 5 were not met (❌)
  • 3 were not applicable (N/A)
Item Recommendation Met by study? Evidence
Objectives
1.1 Purpose of the model Explain the background and objectives for the model ✅ Fully 1 Introduction: “managing COVID-19 at a regional model… discrete-event simulation (DES) of intensive care unit (ICU) patient flow that caters for reconfiguration of ICU wards in runtime” - with overall purpose being to combine that with an agent-based simulation “that predicts the spread of infections in specified area”, feeding that into the DES. Anagnostou et al. (2022)
1.2 Model outputs Define all quantitative performance measures that are reported, using equations where necessary. Specify how and when they are calculated during the model run along with how any measures of error such as confidence intervals are calculated. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): “These include outputs of every replication as well as averages and 95% confidence intervals of the simulation run. Output data includes beds availability and capacity per type, beds availability and capacity per zone, 7-day moving average COVID-19 occupancy, cumulative number of patients moved due to bed shortages for each type, cumulative number of patients discharged per type, cumulative patients died per type.” Anagnostou et al. (2022)
1.3 Experimentation aims If the model has been used for experimentation, state the objectives that it was used to investigate.
(A) Scenario based analysis – Provide a name and description for each scenario, providing a rationale for the choice of scenarios and ensure that item 2.3 (below) is completed.
(B) Design of experiments – Provide details of the overall design of the experiments with reference to performance measures and their parameters (provide further details in data below).
(C) Simulation Optimisation – (if appropriate) Provide full details of what is to be optimised, the parameters that were included and the algorithm(s) that was be used. Where possible provide a citation of the algorithm(s).		 N/A No experimentation - just presents results from one run of the model with a single set of parameters.
Logic
2.1 Base model overview diagram Describe the base model using appropriate diagrams and description. This could include one or more process flow, activity cycle or equivalent diagrams sufficient to describe the model to readers. Avoid complicated diagrams in the main text. The goal is to describe the breadth and depth of the model with respect to the system being studied. ✅ Fully Figure 2: “CHARM patient flow diagram” Anagnostou et al. (2022)
2.2 Base model logic Give details of the base model logic. Give additional model logic details sufficient to communicate to the reader how the model works. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): “The model logic is shown in Figure 2 and applies on every patient that enters the model. There are three types of arrivals for C, EM and EL patients. When there is a new arrival, the model checks whether there is bed availability in the respective wards. If there is no bed available, the patient is moved out of the specific facility. If a bed is available… [continues]” Anagnostou et al. (2022)
2.3 Scenario logic Give details of the logical difference between the base case model and scenarios (if any). This could be incorporated as text or where differences are substantial could be incorporated in the same manner as 2.2. N/A No scenarios
2.4 Algorithms Provide further detail on any algorithms in the model that (for example) mimic complex or manual processes in the real world (i.e. scheduling of arrivals/ appointments/ operations/ maintenance, operation of a conveyor system, machine breakdowns, etc.). Sufficient detail should be included (or referred to in other published work) for the algorithms to be reproducible. Pseudo-code may be used to describe an algorithm. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): “Patient arrivals and LoS are sampled by triangular distributions. Mortality rate is a probability (a number between 0-1) and occupancy thresholds are percentages (a number between 0-100). Example input parameter values can be seen in Table 1.” Anagnostou et al. (2022)
2.5.1 Components - entities Give details of all entities within the simulation including a description of their role in the model and a description of all their attributes. ✅ Fully Presented in Figure 2 and described in 3.2 dynamiC Hospital wARD Management (CHARM). The entities are patients and they can either be COVID-19 (C), emergency (EM), or elective (EL).
2.5.2 Components - activities Describe the activities that entities engage in within the model. Provide details of entity routing into and out of the activity. ✅ Fully Presented in Figure 2 and described in 3.2 dynamiC Hospital wARD Management (CHARM). Activities are to stay in an ICU bed, and to stay in a recovery bed.
2.5.3 Components - resources List all the resources included within the model and which activities make use of them. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): “Hospital wards in CHARM are allocated in zones that represent six types, i.e., COVID-19 (C) and COVID-19 recovery (CR), emergency (EM) and emergency recovery (EMR), and elective (EL) and elective recovery (ELR). These zones are the pool of bed resources for each type. Each ward is dynamically recharacterized as of different zone type according to bed occupancy.” Anagnostou et al. (2022)
2.5.4 Components - queues Give details of the assumed queuing discipline used in the model (e.g. First in First Out, Last in First Out, prioritisation, etc.). Where one or more queues have a different discipline from the rest, provide a list of queues, indicating the queuing discipline used for each. If reneging, balking or jockeying occur, etc., provide details of the rules. Detail any delays or capacity constraints on the queues. ✅ Fully Presented in Figure 2 and described in 3.2 dynamiC Hospital wARD Management (CHARM). If a resource is unavailable, it describes that instead of there being a queue, the patient is simply moved to another hospital (hence, exiting the model).
2.5.5 Components - entry/exit points Give details of the model boundaries i.e. all arrival and exit points of entities. Detail the arrival mechanism (e.g. ‘thinning’ to mimic a non-homogenous Poisson process or balking) ✅ Fully As evident in Figure 2, entry is as a ICU arrival, and then exit is either moving to another hospital, dying, or being discharged.
Data
3.1 Data sources List and detail all data sources. Sources may include:
• Interviews with stakeholders,
• Samples of routinely collected data,
• Prospectively collected samples for the purpose of the simulation study,
• Public domain data published in either academic or organisational literature. Provide, where possible, the link and DOI to the data or reference to published literature.
All data source descriptions should include details of the sample size, sample date ranges and use within the study.		 ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): example input values are “derived from Simpson et al. (2005) and Melman, Parlikad and Cameron (2021).” Anagnostou et al. (2022)
3.2 Pre-processing Provide details of any data manipulation that has taken place before its use in the simulation, e.g. interpolation to account for missing data or the removal of outliers. N/A None mentioned, and cannot presume whether there might have been any or not.
3.3 Input parameters List all input variables in the model. Provide a description of their use and include parameter values. For stochastic inputs provide details of any continuous, discrete or empirical distributions used along with all associated parameters. Give details of all time dependent parameters and correlation.
Clearly state:
• Base case data
• Data use in experimentation, where different from the base case.
• Where optimisation or design of experiments has been used, state the range of values that parameters can take.
• Where theoretical distributions are used, state how these were selected and prioritised above other candidate distributions.		 ✅ Fully Provided in Table 1
3.4 Assumptions Where data or knowledge of the real system is unavailable what assumptions are included in the model? This might include parameter values, distributions or routing logic within the model. ❌ Not met -
Experimentation
4.1 Initialisation Report if the system modelled is terminating or non-terminating. State if a warm-up period has been used, its length and the analysis method used to select it. For terminating systems state the stopping condition.
State what if any initial model conditions have been included, e.g., pre-loaded queues and activities. Report whether initialisation of these variables is deterministic or stochastic.		 ❌ Not met -
4.2 Run length Detail the run length of the simulation model and time units. 🟡 Partially Mentions time unit (e.g. “daily ICU arrivals” in Figure 2), but not run length. Anagnostou et al. (2022)
4.3 Estimation approach State the method used to account for the stochasticity: For example, two common methods are multiple replications or batch means. Where multiple replications have been used, state the number of replications and for batch means, indicate the batch length and whether the batch means procedure is standard, spaced or overlapping. For both procedures provide a justification for the methods used and the number of replications/size of batches. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): mentions that you should input “the number of replications that we wish to run the simulation for”. Not applicable to justify number of replications in the context of this paper, since it is just an example of inputs to the model.
Implementation
5.1 Software or programming language State the operating system and version and build number.
State the name, version and build number of commercial or open source DES software that the model is implemented in.
State the name and version of general-purpose programming languages used (e.g. Python 3.5).
Where frameworks and libraries have been used provide all details including version numbers.		 🟡 Partially Just mentions language and package - 3.2 dynamiC Hospital wARD Management (CHARM): “CHARM is built in Python using the SimPy libraries”. Anagnostou et al. (2022)
5.2 Random sampling State the algorithm used to generate random samples in the software/programming language used e.g. Mersenne Twister.
If common random numbers are used, state how seeds (or random number streams) are distributed among sampling processes.		 ❌ Not met -
5.3 Model execution State the event processing mechanism used e.g. three phase, event, activity, process interaction.
Note that in some commercial software the event processing mechanism may not be published. In these cases authors should adhere to item 5.1 software recommendations.
State all priority rules included if entities/activities compete for resources.
If the model is parallel, distributed and/or use grid or cloud computing, etc., state and preferably reference the technology used. For parallel and distributed simulations the time management algorithms used. If the HLA is used then state the version of the standard, which run-time infrastructure (and version), and any supporting documents (FOMs, etc.)		 ❌ Not met -
5.4 System specification State the model run time and specification of hardware used. This is particularly important for large scale models that require substantial computing power. For parallel, distributed and/or use grid or cloud computing, etc. state the details of all systems used in the implementation (processors, network, etc.) ❌ Not met -
Code access
6.1 Computer model sharing statement Describe how someone could obtain the model described in the paper, the simulation software and any other associated software (or hardware) needed to reproduce the results. Provide, where possible, the link and DOIs to these. ✅ Fully 4 FACS-CHARM hybrid model: “The CHARM code is available on a public repository (https://gitlab.com/anabrunel/charm). We have also developed a dashboard CHARM that enables end users to enter input parameters in a user-friendly manner and visualize the output plots. This code is also publicly available at https://gitlab.com/anabrunel/charm-app. The CHARM dashboard is deployed on a demo server available at https://charm-des-app.herokuapp.com/.” Anagnostou et al. (2022)

DES checklist derived from ISPOR-SDM

Of the 18 items in the checklist:

  • 8 were met fully (✅)
  • 3 were partially met (🟡)
  • 4 were not met (❌)
  • 3 were not applicable (N/A)
Item Assessed if… Met by study? Evidence/location
Model conceptualisation
1 Is the focused health-related decision problem clarified? …the decision problem under investigation was defined. DES studies included different types of decision problems, eg, those listed in previously developed taxonomies. ✅ Fully 1 Introduction: They aim to “support decision-making for COVID-19”, developing a “discrete-event simulation (DES) of intensive care unit (ICU) patient flow that caters for reconfiguration of ICU wards in runtime.” This is combine with estimates of spread of COVID-19 infections to then be used for “managing COVID-19 at a regional level”. “The Hybrid FACS-CHARM model can be used to explore questions regarding public health interventions and their impact on the local hospital facilities. Example questions are… [continues]” Anagnostou et al. (2022)
2 Is the modeled healthcare setting/health condition clarified? …the physical context/scope (eg, a certain healthcare unit or a broader system) or disease spectrum simulated was described. ✅ Fully In 1 Introduction, they provide context on the COVID-19 pandemic in - e.g. “At the date of writing (May 2022) the World Health Organization (WHO) recorded more that 500 million cases world-wide and more than six million recorded deaths (WHO 2022a).” They also describe the general context in terms of decision-making for COVID-19 - e.g. “National and local governments have implemented various measures at various points in time according to the severity of the disease in that location, the economic constraints and inputs from experts… [continues]” Anagnostou et al. (2022)
3 Is the model structure described? …the model’s conceptual structure was described in the form of either graphical or text presentation. ✅ Fully Presented in Figure 2 and described in 3.2 dynamiC Hospital wARD Management (CHARM): “The model logic is shown in Figure 2 and applies on every patient that enters the model. There are three types of arrivals for C, EM and EL patients. When there is a new arrival, the model checks whether there is bed availability in the respective wards. If there is no bed available, the patient is moved out of the specific facility. If a bed is available… [continues]” Anagnostou et al. (2022)
4 Is the time horizon given? …the time period covered by the simulation was reported. ❌ Not met -
5 Are all simulated strategies/scenarios specified? …the comparators under test were described in terms of their components, corresponding variations, etc N/A No scenarios - just runs with a single set of input parameters.
6 Is the target population described? …the entities simulated and their main attributes were characterized. 🟡 Partially In the introduction, they describe COVID-19 cases worldwide (e.g. “500 million cases world-wide and more than six million recorded deaths”). However, no description is provided of ICU units dealing with COVID-19 cases, which is the focus of the model. Anagnostou et al. (2022)
Paramaterisation and uncertainty assessment
7 Are data sources informing parameter estimations provided? …the sources of all data used to inform model inputs were reported. ✅ Fully 3.2 dynamiC Hospital wARD Management (CHARM): example input values are “derived from Simpson et al. (2005) and Melman, Parlikad and Cameron (2021).” Anagnostou et al. (2022)
8 Are the parameters used to populate model frameworks specified? …all relevant parameters fed into model frameworks were disclosed. ✅ Fully Provided in Table 1
9 Are model uncertainties discussed? …the uncertainty surrounding parameter estimations and adopted statistical methods (eg, 95% confidence intervals or possibility distributions) were reported. ✅ Fully Presents results as mean and confidence intervals in Figure 3.
10 Are sensitivity analyses performed and reported? …the robustness of model outputs to input uncertainties was examined, for example via deterministic (based on parameters’ plausible ranges) or probabilistic (based on a priori-defined probability distributions) sensitivity analyses, or both. N/A Not met, although this is felt appropriate given the stated scope of the article. 1 Introduction: “The purpose of this paper is to present our approach in using hybrid ABS-DES for COVID-19 management at a regional level and discuss its potential. It is therefore not in the scope of this manuscript to study specific use cases and present experimental results.” Anagnostou et al. (2022)
Validation
11 Is face validity evaluated and reported? …it was reported that the model was subjected to the examination on how well model designs correspond to the reality and intuitions. It was assumed that this type of validation should be conducted by external evaluators with no stake in the study. ❌ Not met -
12 Is cross validation performed and reported …comparison across similar modeling studies which deal with the same decision problem was undertaken. ❌ Not met -
13 Is external validation performed and reported? …the modeler(s) examined how well the model’s results match the empirical data of an actual event modeled. ❌ Not met -
14 Is predictive validation performed or attempted? …the modeler(s) examined the consistency of a model’s predictions of a future event and the actual outcomes in the future. If this was not undertaken, it was assessed whether the reasons were discussed. N/A Only relevant to forecasting models
Generalisability and stakeholder involvement
15 Is the model generalizability issue discussed? …the modeler(s) discussed the potential of the resulting model for being applicable to other settings/populations (single/multiple application). 🟡 Partially I did not spot anywhere where they explicitly address this, but we get a sense of its potential generalisability to different countries from 5 Conclusions: “Currently, within the STAMINA project, we are developing three cases studies in Turkey, Romania and Lithuania.”
16 Are decision makers or other stakeholders involved in modeling? …the modeler(s) reported in which part throughout the modeling process decision makers and other stakeholders (eg, subject experts) were engaged. ✅ Fully Acknowledgements: “The initial idea of developing a hybrid ABS-DES model for regional COVID-19 prediction was formed at the start of the Coronavirus pandemic in the UK in March 2020. The concept was discussed in brainstorming sessions by Dr Anastasia Anagnostou, Dr Derek Groen, Professor Simon J.E. Taylor, Dr Imran Mahmood, Dr Alaa Marshan, Dr Isabel Sassoon, Dr Alan Serrano, Professor Panos Louvieris and Dr David Bell from Brunel University London.” Anagnostou et al. (2022)
17 Is the source of funding stated? …the sponsorship of the study was indicated. ✅ Fully Acknowledgements: “This work was partially funded by the EU H2020 STAMINA project No. 883441” Anagnostou et al. (2022)
18 Are model limitations discussed? …limitations of the assessed model, especially limitations of interest to decision makers, were discussed. 🟡 Partially Very little discussion, just mentions one limitation - 5 Conclusions: “One of the limitations of the model is that we include only ICU bed resources”

References

Anagnostou, Anastasia, Derek Groen, Simon J. E. Taylor, Diana Suleimenova, Nura Abubakar, Arindam Saha, Kate Mintram, et al. 2022. “FACS-CHARM: A Hybrid Agent-Based and Discrete-Event Simulation Approach for Covid-19 Management at Regional Level.” In 2022 Winter Simulation Conference (WSC), 1223–34. https://doi.org/10.1109/WSC57314.2022.10015462.
Monks, Thomas, Christine S. M. Currie, Bhakti Stephan Onggo, Stewart Robinson, Martin Kunc, and Simon J. E. Taylor. 2019. “Strengthening the Reporting of Empirical Simulation Studies: Introducing the STRESS Guidelines.” Journal of Simulation 13 (1): 55–67. https://doi.org/10.1080/17477778.2018.1442155.
Zhang, Xiange, Stefan K. Lhachimi, and Wolf H. Rogowski. 2020. “Reporting Quality of Discrete Event Simulations in Healthcare—Results From a Generic Reporting Checklist.” Value in Health 23 (4): 506–14. https://doi.org/10.1016/j.jval.2020.01.005.