Summary report

For computational reproducibility assessment of Huang et al. 2019

Study

Huang S, Maingard J, Kok HK, Barras CD, Thijs V, Chandra RV, Brooks DM and Asadi H. Optimizing Resources for Endovascular Clot Retrieval for Acute Ischemic Stroke, a Discrete Event Simulation. Frontiers in Neurology 10, 653 (2019). https://doi.org/10.3389/fneur.2019.00653.

This is a discrete-event simulation model of an endovascular clot retrieval (ECR) service. ECR is a treatment for acute ischaemic stroke. The model includes the stroke pathway, as well as three other pathways that share resources with the stroke pathway: an elective non-stroke interventional neuroradiology pathway, an emergency interventional radiology pathway, and an elective interventional radiology pathway. The model is created using R Simmer. The paper explores waiting times and resource utilisation - particularly focussing on the biplane angiographic suite (angioINR). A few scenarios are tried to help examine why the wait times are so high for the angioINR. Images from the original study on this page are sourced from Huang et al. (2019).

Computational reproducibility

Successfully reproduced 37.5% (3 out of 8) of items from the scope in 24h 10m (60.4%).

Required troubleshooting:

• Environment - identifying and installing required packages (spent some time trying to use estimated versions, but ended up using latest)
• Extract model code - was set within code for shiny app
• Get model parameters - parameters differed between code and paper
• Writing code to implement and run scenarios
• Writing code to process results and produce figures - which took a bit of time, since there were several transformations that were complicated or not mentioned
• Seeds - introduced as results varied a fair amount between runs

Figure 2

Consensus: Not reproduced

Figure 3

Consensus: Not reproduced

Figure 4

Consensus: Not reproduced

Figure 5

Consensus: Successfully reproduced

Supplementary figure

Consensus: Not reproduced

In-text result 1

Consensus: Successfully reproduced

“Exclusive-Use Scenario. In this scenario, the overall wait time probability at angioINR was reduced compared to baseline (red line in Figure 2B compared to Figure 2A). This represents a decrease in ECR patient wait time for angioINR by an average of 6 min.” Huang et al. (2019)

Reproduction:

	scenario	mean	diff_from_baseline
0	Baseline	13.958269	0.00
1	Exclusive use	8.117729	-5.84

In-text result 2

Consensus: Successfully reproduced

“Two angioINRs Scenario. This scenario simulates the effect a facility upgrade to two biplane angiographic suites, but without additional staff changes. The wait time probability at angioINR was reduced compared to baseline (Figure 2C). The reduction represents an average of 4 min less in queue for angioINR.” Huang et al. (2019)

Reproduction:

	scenario	mean	diff_from_baseline
0	Baseline	13.958269	0.00
2	Two AngioINRs	9.621122	-4.34

In-text result 3

Consensus: Not reproduced

“Extended Schedule Scenario. The wait time probability at angioINR in the exclusive- use scenario was further reduced by extended work hours (Figure 3B). In contrast, work extension did not affect baseline or the 2 angioINRs scenario (Figures 3A,C). For the baseline scenario, 1 and 2 h of extra work resulted in an average wait time of 1.7 and 0.9 min reduction, respectively. For the 2 angioINRs scenario, 1 and 2 h of extra work resulted in an average wait time gain of 1 and 0.3 min, respectively.” Huang et al. (2019)

Reproduction:

	scenario	shift	mean	diff_from_5pm
0	Baseline	5pm	13.958269	0.00
1	Baseline	6pm	12.486042	-1.47
2	Baseline	7pm	12.491421	-1.47
6	Two AngioINRs	5pm	9.621122	0.00
7	Two AngioINRs	6pm	9.216435	-0.40
8	Two AngioINRs	7pm	8.699223	-0.92

Evaluation against guidelines

^{Context: The original study repository was evaluated against criteria from journal badges relating to how open and reproducible the model is and against guidance for sharing artefacts from the STARS framework. The original study article and supplementary materials (excluding code) were evaluated against reporting guidelines for DES models: STRESS-DES, and guidelines adapted from ISPOR-SDM.}

References

Huang, Shiwei, Julian Maingard, Hong Kuan Kok, Christen D. Barras, Vincent Thijs, Ronil V. Chandra, Duncan Mark Brooks, and Hamed Asadi. 2019. “Optimizing Resources for Endovascular Clot Retrieval for Acute Ischemic Stroke, a Discrete Event Simulation.” Frontiers in Neurology 10 (June). https://doi.org/10.3389/fneur.2019.00653.