Reporting guidelines

This page evaluates the extent to which the journal article meets the criteria from two discrete-event simulation study reporting guidelines:

STRESS-DES

Of the 24 items in the checklist:

  • 15 were met fully (✅)
  • 3 were partially met (🟡)
  • 3 were not met (❌)
  • 3 were not applicable (N/A)
Item Recommendation Met by study? Evidence
Objectives
1.1 Purpose of the model Explain the background and objectives for the model ✅ Fully Provided in 1. Introduction -“The International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) has recently formed a Taskforce on COVID-19 to provide guidance to laboratory practitioners in managing this challenge [COVID-19]… A global survey by the Taskforce has then revealed that clinical laboratories have used PPE variably. The laboratories also found it challenging to manage staff rostering, split team arrangement and maintain workplace social distancing (physical distancing)… this simulation study was conducted to explore the relative impact of staff rostering, split team arrangement, social distancing and use of PPE on the potential risk of transmission within the laboratory environment. From the results of this simulation, several recommendations are de- veloped to further assist laboratories in planning their workplace in order to minimize the risk of transmission of SARS-Cov-2 infection.”

Lim et al. (2020)
1.2 Model outputs Define all quantitative performance measures that are reported, using equations where necessary. Specify how and when they are calculated during the model run along with how any measures of error such as confidence intervals are calculated. ✅ Fully All outputs are medians. 2.2 Transmission assumptions - “the median of the proportion of staff infected are recorded”

Lim et al. (2020)
1.3 Experimentation aims If the model has been used for experimentation, state the objectives that it was used to investigate.
(A) Scenario based analysis – Provide a name and description for each scenario, providing a rationale for the choice of scenarios and ensure that item 2.3 (below) is completed.
(B) Design of experiments – Provide details of the overall design of the experiments with reference to performance measures and their parameters (provide further details in data below).
(C) Simulation Optimisation – (if appropriate) Provide full details of what is to be optimised, the parameters that were included and the algorithm(s) that was be used. Where possible provide a citation of the algorithm(s).		 ✅ Fully The various parameters and scenarios of the analysis are described in detail in 2. Material and methods in sections:
• 2.1.1 Number of staff per shift and number of shifts
• 2.1.2 Overall number of staff accessible to work in the laboratory (i.e. overall staff pool)
• 2.1.3 Shift arrangement
• 2.1.4 Split team arrangement
• 2.3 Secondary attack rate
• 2.4 Impact of personal protective equipment
• 2.5 Impact of social distancing
Logic
2.1 Base model overview diagram Describe the base model using appropriate diagrams and description. This could include one or more process flow, activity cycle or equivalent diagrams sufficient to describe the model to readers. Avoid complicated diagrams in the main text. The goal is to describe the breadth and depth of the model with respect to the system being studied. ✅ Fully Figure 1

Figure 1

Lim et al. (2020)
2.2 Base model logic Give details of the base model logic. Give additional model logic details sufficient to communicate to the reader how the model works. ✅ Fully Detailed in 2. Material and methods and supported by Figure 1.
2.3 Scenario logic Give details of the logical difference between the base case model and scenarios (if any). This could be incorporated as text or where differences are substantial could be incorporated in the same manner as 2.2. ✅ Fully Detailed in 2. Material and methods
2.4 Algorithms Provide further detail on any algorithms in the model that (for example) mimic complex or manual processes in the real world (i.e. scheduling of arrivals/ appointments/ operations/ maintenance, operation of a conveyor system, machine breakdowns, etc.). Sufficient detail should be included (or referred to in other published work) for the algorithms to be reproducible. Pseudo-code may be used to describe an algorithm. ✅ Fully The only sampling algorithm used in the model is the Poisson distribution for sampling the number of successful contact per infected staff, and this is described in 2.2 Transmission assumptions

“Due to the stochastic nature of virus transmission, not all contacts lead to successful virus transmissions. Therefore, the probabilistic factor of p is applied to the average contact rate parameter c in a modified Poisson distribution for the number of “successful” contacts, which refers to contacts that lead to successful viral transmission, per work shift. Mathematically, it is described by: P(k) = (pc)k/k! e-pc where P(k) represents the probability for k successful contact in a work shift, with k being a non-negative integer, p is the probability of transmission while c is the average contact rate (i.e. average number of unique contacts in a shift). From this Poisson distribution k is drawn randomly when susceptible staff are assigned to a shift with at least one infected staff. Subsequently, k staff are randomly drawn from other staff in the same shift to be infected to propagate the infection. Of note, P is a function of R0 (the basic reproduction number), contact rate and length of infection period for an entire population”

Lim et al. (2020)
2.5.1 Components - entities Give details of all entities within the simulation including a description of their role in the model and a description of all their attributes. ✅ Fully 2. Material and methods - “The entities in the simulation are laboratory staff assigned to work in a particular shift.”

Lim et al. (2020)
2.5.2 Components - activities Describe the activities that entities engage in within the model. Provide details of entity routing into and out of the activity. ✅ Fully The activity in the model is to be either on or off shift. 2. Material and methods - “In this simple model, each staff can only assume one of the two states, namely staying at home or working in the laboratory. The state of working in the laboratory is further divided into sub-states, representing each work shift, for rosters with more than 1 shift per day.”

Lim et al. (2020)
2.5.3 Components - resources List all the resources included within the model and which activities make use of them. N/A The entity (lab staff) does not require any resources to complete the activity (going on shift). Instead, the model just requires a certain number of entities to be doing the activity.
2.5.4 Components - queues Give details of the assumed queuing discipline used in the model (e.g. First in First Out, Last in First Out, prioritisation, etc.). Where one or more queues have a different discipline from the rest, provide a list of queues, indicating the queuing discipline used for each. If reneging, balking or jockeying occur, etc., provide details of the rules. Detail any delays or capacity constraints on the queues. N/A There are no queues for activities.
2.5.5 Components - entry/exit points Give details of the model boundaries i.e. all arrival and exit points of entities. Detail the arrival mechanism (e.g. ‘thinning’ to mimic a non-homogenous Poisson process or balking) ✅ Fully Entities remain in the model for the whole duration.

2.2 Transmission assumptions - “Throughout the simulation, it is assumed that the infected staff are not quarantined”

Lim et al. (2020)
Data
3.1 Data sources List and detail all data sources. Sources may include:
• Interviews with stakeholders,
• Samples of routinely collected data,
• Prospectively collected samples for the purpose of the simulation study,
• Public domain data published in either academic or organisational literature. Provide, where possible, the link and DOI to the data or reference to published literature.
All data source descriptions should include details of the sample size, sample date ranges and use within the study.		 ✅ Fully 2.1 Workplace assumptions - “The workplace assumptions are arbitrarily determined to represent a wide range of laboratory scenarios.”

4.3 Other observations - “These data on protective effects of PPE are obtained from a meta-analysis of 6 case-control studies related to severe acute respiratory syndrome (SARS) [17], but should be broadly applicable to the current COVID-19 situation.” - although [17] does not have this information, it appears this intends to cite [18] in their references, Jefferson T, Dooley L, Ferroni E, Al-Ansary LA, van Driel ML, Bawazeer GA, Jones MA, Hoffmann TC, Clark J, Beller EM, Glasziou PP, Conly JM. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. 2023 Jan 30;1(1):CD006207. doi: 10.1002/14651858.CD006207.pub6. PMID: 36715243; PMCID: PMC9885521.

Lim et al. (2020)
3.2 Pre-processing Provide details of any data manipulation that has taken place before its use in the simulation, e.g. interpolation to account for missing data or the removal of outliers. N/A Based on data sources, assume this to be not applicable.
3.3 Input parameters List all input variables in the model. Provide a description of their use and include parameter values. For stochastic inputs provide details of any continuous, discrete or empirical distributions used along with all associated parameters. Give details of all time dependent parameters and correlation.
Clearly state:
• Base case data
• Data use in experimentation, where different from the base case.
• Where optimisation or design of experiments has been used, state the range of values that parameters can take.
• Where theoretical distributions are used, state how these were selected and prioritised above other candidate distributions.		 ✅ Fully Nearly all parameters are included in Table 1

Table 1

The others are described in the article though, e.g. 2.5 Impact of social distancing - “To simulate the effect of workplace social distancing, c is arbitrarily reduced by halve”

Lim et al. (2020)
3.4 Assumptions Where data or knowledge of the real system is unavailable what assumptions are included in the model? This might include parameter values, distributions or routing logic within the model. ✅ Fully Extensively details assumptions throughout 2. Material and methods and 4 Discussions
Experimentation
4.1 Initialisation Report if the system modelled is terminating or non-terminating. State if a warm-up period has been used, its length and the analysis method used to select it. For terminating systems state the stopping condition.
State what if any initial model conditions have been included, e.g., pre-loaded queues and activities. Report whether initialisation of these variables is deterministic or stochastic.		 ❌ Not met Not described.
4.2 Run length Detail the run length of the simulation model and time units. ✅ Fully Time unit is days (1, 2 or 3 shifts per day), and runs for 21 days. Described in article, e.g. 2.2 Transmission assumptions - “duration of simulation (i.e. 21 days)”

Lim et al. (2020)
4.3 Estimation approach State the method used to account for the stochasticity: For example, two common methods are multiple replications or batch means. Where multiple replications have been used, state the number of replications and for batch means, indicate the batch length and whether the batch means procedure is standard, spaced or overlapping. For both procedures provide a justification for the methods used and the number of replications/size of batches. 🟡 Partially States number of replications but does not provide justification

2.2 Transmission assumptions - “In view of the stochastic nature of virus transmission and roster allocation, simulations with the same parameters are repeated 100 times and the median of the proportion of staff infected are recorded.”

Lim et al. (2020)
Implementation
5.1 Software or programming language State the operating system and version and build number.
State the name, version and build number of commercial or open source DES software that the model is implemented in.
State the name and version of general-purpose programming languages used (e.g. Python 3.5).
Where frameworks and libraries have been used provide all details including version numbers.		 🟡 Partially 2.7 Simulation package - “This simulation was performed with codes written in Python 3 on a desktop computer (Intel Core i5 3.5 GHz, 8 GB RAM). Standard libraries such as NumPy and pandas are employed.”

Does not mention operating system, or version of the packages used.

Lim et al. (2020)
5.2 Random sampling State the algorithm used to generate random samples in the software/programming language used e.g. Mersenne Twister.
If common random numbers are used, state how seeds (or random number streams) are distributed among sampling processes.		 ❌ Not met Not described in the paper. Know from the code that they used common random numbers without seed control.
5.3 Model execution State the event processing mechanism used e.g. three phase, event, activity, process interaction.
Note that in some commercial software the event processing mechanism may not be published. In these cases authors should adhere to item 5.1 software recommendations.
State all priority rules included if entities/activities compete for resources.
If the model is parallel, distributed and/or use grid or cloud computing, etc., state and preferably reference the technology used. For parallel and distributed simulations the time management algorithms used. If the HLA is used then state the version of the standard, which run-time infrastructure (and version), and any supporting documents (FOMs, etc.)		 ❌ Not met Not described
5.4 System specification State the model run time and specification of hardware used. This is particularly important for large scale models that require substantial computing power. For parallel, distributed and/or use grid or cloud computing, etc. state the details of all systems used in the implementation (processors, network, etc.) 🟡 Partially 2.7 Simulation package - “This simulation was performed with codes written in Python 3 on a desktop computer (Intel Core i5 3.5 GHz, 8 GB RAM).”

Does not mention run time

Lim et al. (2020)
Code access
6.1 Computer model sharing statement Describe how someone could obtain the model described in the paper, the simulation software and any other associated software (or hardware) needed to reproduce the results. Provide, where possible, the link and DOIs to these. ✅ Fully 2.7 Simulation package - “The simulation codes used in this study are provided here (https://github.com/chaose5/COVID- roster-simulation) and in the Supplemental Material.”

Lim et al. (2020)

DES checklist derived from ISPOR-SDM

Of the 18 items in the checklist:

  • 12 were met fully (✅)
  • 4 were not met (❌)
  • 2 were not applicable (N/A)
Item Assessed if… Met by study? Evidence/location
Model conceptualisation
1 Is the focused health-related decision problem clarified? …the decision problem under investigation was defined. DES studies included different types of decision problems, eg, those listed in previously developed taxonomies. ✅ Fully Minimising transmission of COVID-19 in laboratories, as in 1 Introduction.
2 Is the modeled healthcare setting/health condition clarified? …the physical context/scope (eg, a certain healthcare unit or a broader system) or disease spectrum simulated was described. ✅ Fully The context here is the laboratories and their strategies to minimise transmission. It describes the current situation, with regards to that. Other description of the laboratories is not felt to be necessary.

1 Introduction - “The International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) has recently formed a Taskforce on COVID-19 to provide guidance to aboratory practitioners in managing this challenge. The Task Force has already published biosafety recommendations, which outlined the steps that laboratories operating at biosafety level 1 and level 2 shall or may use to lower the risk of workplace transmission of the virus. These in- cluded the use of personal protective equipment (PPE), temperature and symptom monitoring, split team work arrangements and workplace social distancing. A global survey by the Taskforce has then revealed that clinical laboratories have used PPE variably. The laboratories also found it challenging to manage staff rostering, split team arrangement and maintain workplace social distancing (physical distancing)” Lim et al. (2020)
3 Is the model structure described? …the model’s conceptual structure was described in the form of either graphical or text presentation. ✅ Fully Detailed in 2. Material and methods and supported by Figure 1.

Figure 1

Lim et al. (2020)
4 Is the time horizon given? …the time period covered by the simulation was reported. ✅ Fully 2.2 Transmission assumptions - “duration of simulation (i.e. 21 days)”

Lim et al. (2020)
5 Are all simulated strategies/scenarios specified? …the comparators under test were described in terms of their components, corresponding variations, etc ✅ Fully The various parameters and scenarios of the analysis are described in detail in 2. Material and methods in sections:
• 2.1.1 Number of staff per shift and number of shifts
• 2.1.2 Overall number of staff accessible to work in the laboratory (i.e. overall staff pool)
• 2.1.3 Shift arrangement
• 2.1.4 Split team arrangement
• 2.3 Secondary attack rate
• 2.4 Impact of personal protective equipment
• 2.5 Impact of social distancing
6 Is the target population described? …the entities simulated and their main attributes were characterized. ✅ Fully Population being modelled is laboratory staff and the transmission of COVID-19 between them. Given there would likely be little data on transmission of COVID between staff at the point of publication, it is felt the population is sufficiently, with the main description being the results from a survey of attitudes towards PPE use. 1 Introduction - “A global survey by the Taskforce has then revealed that clinical laboratories have used PPE variably. The laboratories also found it challenging to manage staff rostering, split team arrangement and maintain workplace social distancing (physical distancing)” Lim et al. (2020)
Paramaterisation and uncertainty assessment
7 Are data sources informing parameter estimations provided? …the sources of all data used to inform model inputs were reported. ✅ Fully 2.1 Workplace assumptions - “The workplace assumptions are arbitrarily determined to represent a wide range of laboratory scenarios.”

4.3 Other observations - “These data on protective effects of PPE are obtained from a meta-analysis of 6 case-control studies related to severe acute respiratory syndrome (SARS) [17], but should be broadly applicable to the current COVID-19 situation.” - although [17] does not have this information, it appears this intends to cite [18] in their references, Jefferson T, Dooley L, Ferroni E, Al-Ansary LA, van Driel ML, Bawazeer GA, Jones MA, Hoffmann TC, Clark J, Beller EM, Glasziou PP, Conly JM. Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev. 2023 Jan 30;1(1):CD006207. doi: 10.1002/14651858.CD006207.pub6. PMID: 36715243; PMCID: PMC9885521.

Lim et al. (2020)
8 Are the parameters used to populate model frameworks specified? …all relevant parameters fed into model frameworks were disclosed. ✅ Fully Nearly all parameters are included in Table 1

Table 1

The others are described in the article though, e.g. 2.5 Impact of social distancing - “To simulate the effect of workplace social distancing, c is arbitrarily reduced by halve”

Lim et al. (2020)
9 Are model uncertainties discussed? …the uncertainty surrounding parameter estimations and adopted statistical methods (eg, 95% confidence intervals or possibility distributions) were reported. ❌ Not met Just presents median.
10 Are sensitivity analyses performed and reported? …the robustness of model outputs to input uncertainties was examined, for example via deterministic (based on parameters’ plausible ranges) or probabilistic (based on a priori-defined probability distributions) sensitivity analyses, or both. ✅ Fully Conducts sensitivty analyses, varying the model parameters to explore impact on results, e.g. 3.1 General staff roster arrangement - “The trend of the results of the baseline scenario (15% secondary attach rate) were reproduced in the sensitivity analysis using different sec- ondary attack rates of 5% and 30%.”

Lim et al. (2020)
Validation
11 Is face validity evaluated and reported? …it was reported that the model was subjected to the examination on how well model designs correspond to the reality and intuitions. It was assumed that this type of validation should be conducted by external evaluators with no stake in the study. ❌ Not met -
12 Is cross validation performed and reported …comparison across similar modeling studies which deal with the same decision problem was undertaken. ❌ Not met -
13 Is external validation performed and reported? …the modeler(s) examined how well the model’s results match the empirical data of an actual event modeled. N/A Since this article was submit relatively early in the COVID-19 pandemic, it is reasonably assumed that there may not yet have been empirical data available to compare against, although we note that the paper does not mention whether or not this is the case.
14 Is predictive validation performed or attempted? …the modeler(s) examined the consistency of a model’s predictions of a future event and the actual outcomes in the future. If this was not undertaken, it was assessed whether the reasons were discussed. N/A This is only relevant to forecasting models.
Generalisability and stakeholder involvement
15 Is the model generalizability issue discussed? …the modeler(s) discussed the potential of the resulting model for being applicable to other settings/populations (single/multiple application). ✅ Fully 2.1 Workplace assumptions - “The workplace assumptions are arbitrarily determined to represent a wide range of laboratory scenarios.”

5. Conclusion - “The several broad recommendations are drawn from the results of the simulations. The recommendations are not meant to be pre- scriptive. Each laboratory operates in a unique environment and should tailor their practices to best suit their priorities within the available resources.”

Lim et al. (2020)
16 Are decision makers or other stakeholders involved in modeling? …the modeler(s) reported in which part throughout the modeling process decision makers and other stakeholders (eg, subject experts) were engaged. ❌ Not met -
17 Is the source of funding stated? …the sponsorship of the study was indicated. ✅ Fully Acknowledgement - “Sources of support None.”

Lim et al. (2020)
18 Are model limitations discussed? …limitations of the assessed model, especially limitations of interest to decision makers, were discussed. ✅ Fully Assumptions and limitations of the model and study are described in 2. Material and methods and 4 Discussion

References

Lim, Chun Yee, Mary Kathryn Bohn, Giuseppe Lippi, Maurizio Ferrari, Tze Ping Loh, Kwok-Yung Yuen, Khosrow Adeli, and Andrea Rita Horvath. 2020. “Staff Rostering, Split Team Arrangement, Social Distancing (Physical Distancing) and Use of Personal Protective Equipment to Minimize Risk of Workplace Transmission During the COVID-19 Pandemic: A Simulation Study.” Clinical Biochemistry 86 (December): 15–22. https://doi.org/10.1016/j.clinbiochem.2020.09.003.
Monks, Thomas, Christine S. M. Currie, Bhakti Stephan Onggo, Stewart Robinson, Martin Kunc, and Simon J. E. Taylor. 2019. “Strengthening the Reporting of Empirical Simulation Studies: Introducing the STRESS Guidelines.” Journal of Simulation 13 (1): 55–67. https://doi.org/10.1080/17477778.2018.1442155.
Zhang, Xiange, Stefan K. Lhachimi, and Wolf H. Rogowski. 2020. “Reporting Quality of Discrete Event Simulations in Healthcare—Results From a Generic Reporting Checklist.” Value in Health 23 (4): 506–14. https://doi.org/10.1016/j.jval.2020.01.005.