Day 6

reproduce
Author

Amy Heather

Published

September 5, 2024

Note

Troubleshooting table 3. Total time used: 9h 48m (24.5%)

When comparing against table 3 on this page, the results are from Johnson et al. (2021)

09.28-09.38, 10.10-10.19: Running 5 years

Copied and modified Case_Detection_Results_5yrs.Rmd as I had for Case_Detection_Results.Rmd yesterday, and then ran that on the remote machine (8.6 minutes).

Rscript -e "rmarkdown::render('scripts/Case_Detection_Results_5yrs.Rmd')"

I’ve noted that the CEplot-1.png goes into a files/figure-gfm folder. I’ve add some code to change the default save location of images, moved the current images, and altered the section names.

Reflection

Duplication between the two .Rmd files - would be easier if both controlled by a single script that could make the two outputs, else when making changes like this, have to carefully make to both files.

10.20-11.30: Resuming Table 3

Add the 5 year results to Table 3 - still assuming that S0 is S1NoCD from 3 year (not from from 5 year or S2 or S3), as can’t spot an S0 anywhere.

Due to size of table, it was difficult to just look between the two tables, so I converted and simplified Table 3 from the paper into a .csv file, and then appended that so I could calculate the differences.

Although some things are similar (e.g. scenario 1 costs and QALYs), others are very different…

import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

pd.read_csv('sall_compare_to_paper.csv')
Scenario Interval CostpAgent CostpAgent_paper CostDiff CostPerc QALYpAgent QALYpAgent_paper QALYDiff QALYPerc ICER ICER_paper ICERDiff ICERPerc IncrementalNMB IncrementalNMB_paper INMBDiff INMBPerc Ranking Ranking_paper RankDiff RankPerc
0 S0 NaN 2149 2151 -2 -0.10% 12.542 12.546 -0.004 0.00% NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 S1a 3 years 2421 2438 -17 -0.70% 12.559 12.560 -0.001 0.00% 16196.0 19632.0 -3436.0 -17.5% 568.0 444.0 124.0 27.9% 3.0 1.0 2.0 200.0%
2 S1a 5 years 2354 2356 -2 -0.10% 12.555 12.556 -0.001 0.00% 20431.0 19847.0 584.0 2.9% 280.0 312.0 -32.0 -10.3% 9.0 2.0 7.0 350.0%
3 S1b 3 years 2371 2363 8 0.30% 12.545 12.554 -0.009 -0.10% 67189.0 25894.0 41295.0 159.5% -57.0 198.0 -255.0 -128.8% 15.0 6.0 9.0 150.0%
4 S1b 5 years 2279 2296 -17 -0.70% 12.550 12.552 -0.002 0.00% 25911.0 23187.0 2724.0 11.7% 111.0 168.0 -57.0 -33.9% 11.0 8.0 3.0 37.5%
5 S1c 3 years 2361 2386 -25 -1.00% 12.554 12.551 0.003 0.00% 18321.0 46956.0 -28635.0 -61.0% 366.0 15.0 351.0 2 340.0% 8.0 16.0 -8.0 -50.0%
6 S1c 5 years 2316 2313 3 0.10% 12.550 12.550 0.000 0.00% 34018.0 38673.0 -4655.0 -12.0% 73.0 47.0 26.0 55.3% 12.0 15.0 -3.0 -20.0%
7 S2a 3 years 2913 2286 627 27.40% 12.287 12.553 -0.266 -2.10% 13392.0 18908.0 -5516.0 -29.2% 595.0 223.0 372.0 166.8% 2.0 4.0 -2.0 -50.0%
8 S2a 5 years 2858 2246 612 27.20% 12.283 12.551 -0.268 -2.10% 16541.0 17514.0 -973.0 -5.6% 366.0 176.0 190.0 108.0% 7.0 7.0 0.0 0.0%
9 S3a 3 years 3052 2234 818 36.60% 11.243 12.548 -1.305 -10.40% 20355.0 30366.0 -10011.0 -33.0% 263.0 54.0 209.0 387.0% 10.0 14.0 -4.0 -28.6%
10 S3a 5 years 3043 2207 836 37.90% 11.238 12.548 -1.310 -10.40% 13424.0 22636.0 -9212.0 -40.7% 392.0 68.0 324.0 476.5% 6.0 12.0 -6.0 -50.0%
11 S3b 3 years 3145 2292 853 37.20% 11.241 12.553 -1.312 -10.50% 43606.0 18438.0 25168.0 136.5% 40.0 241.0 -201.0 -83.4% 13.0 3.0 10.0 333.3%
12 S3b 5 years 3130 2250 880 39.10% 11.242 12.552 -1.310 -10.40% 16179.0 16251.0 -72.0 -0.4% 483.0 206.0 277.0 134.5% 5.0 5.0 0.0 0.0%
13 S3c 3 years 3105 2256 849 37.60% 11.255 12.550 -1.295 -10.30% 11517.0 23972.0 -12455.0 -52.0% 779.0 114.0 665.0 583.3% 1.0 9.0 -8.0 -88.9%
14 S3c 5 years 3042 2224 818 36.80% 11.241 12.549 -1.308 -10.40% 10874.0 20278.0 -9404.0 -46.4% 516.0 107.0 409.0 382.2% 4.0 10.0 -6.0 -60.0%
15 S3d 3 years 3139 2263 876 38.70% 11.239 12.549 -1.310 -10.40% 55904.0 28245.0 27659.0 97.9% -28.0 86.0 -114.0 -132.6% 14.0 11.0 3.0 27.3%
16 S3d 5 years 3045 2227 818 36.70% 11.228 12.548 -1.320 -10.50% -978806.0 27591.0 -1006397.0 -3 647.6% -154.0 62.0 -216.0 -348.4% 16.0 13.0 3.0 23.1%

11.32-11.36, 13.08-13.10: Trying higher agent numbers

Curious is this is due to the number of base agents, I re-ran with 1e6 base_agents (1 million) instead of 5e5 (500,000). On the remote machine, this took 17.1 + 17.1 = 34.2 minutes.

pd.read_csv('sall_compare_to_paper_1e6.csv')
Scenario Interval CostpAgent CostpAgent_paper CostDiff CostPerc QALYpAgent QALYpAgent_paper QALYDiff QALYPerc ICER ICER_paper ICERDiff ICERPerc IncrementalNMB IncrementalNMB_paper INMBDiff INMBPerc Ranking Ranking_paper RankDiff RankPerc
0 S0 NaN 2154 2151 3 0.1% 12.538 12.546 -0.008 -0.10% NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 S1a 3 years 2430 2438 -8 -0.3% 12.558 12.560 -0.002 0.00% 13791.0 19632.0 -5841.0 -29.8% 726.0 444.0 282.0 63.5% 4.0 1.0 3.0 300.0%
2 S1a 5 years 2355 2356 -1 0.0% 12.560 12.556 0.004 0.00% 9740.0 19847.0 -10107.0 -50.9% 845.0 312.0 533.0 170.8% 2.0 2.0 0.0 0.0%
3 S1b 3 years 2365 2363 2 0.1% 12.550 12.554 -0.004 0.00% 17035.0 25894.0 -8859.0 -34.2% 409.0 198.0 211.0 106.6% 12.0 6.0 6.0 100.0%
4 S1b 5 years 2311 2296 15 0.7% 12.554 12.552 0.002 0.00% 10380.0 23187.0 -12807.0 -55.2% 614.0 168.0 446.0 265.5% 8.0 8.0 0.0 0.0%
5 S1c 3 years 2398 2386 12 0.5% 12.552 12.551 0.001 0.00% 17438.0 46956.0 -29518.0 -62.9% 456.0 15.0 441.0 2 940.0% 9.0 16.0 -7.0 -43.8%
6 S1c 5 years 2300 2313 -13 -0.6% 12.544 12.550 -0.006 0.00% 27839.0 38673.0 -10834.0 -28.0% 119.0 47.0 72.0 153.2% 14.0 15.0 -1.0 -6.7%
7 S2a 3 years 2877 2286 591 25.9% 12.292 12.553 -0.261 -2.10% 7853.0 18908.0 -11055.0 -58.5% 1191.0 223.0 968.0 434.1% 1.0 4.0 -3.0 -75.0%
8 S2a 5 years 2813 2246 567 25.2% 12.287 12.551 -0.264 -2.10% 9106.0 17514.0 -8408.0 -48.0% 642.0 176.0 466.0 264.8% 5.0 7.0 -2.0 -28.6%
9 S3a 3 years 3074 2234 840 37.6% 11.248 12.548 -1.300 -10.40% 7930.0 30366.0 -22436.0 -73.9% 803.0 54.0 749.0 1 387.0% 3.0 14.0 -11.0 -78.6%
10 S3a 5 years 3031 2207 824 37.3% 11.235 12.548 -1.313 -10.50% -80780.0 22636.0 -103416.0 -456.9% -228.0 68.0 -296.0 -435.3% 15.0 12.0 3.0 25.0%
11 S3b 3 years 3187 2292 895 39.0% 11.247 12.553 -1.306 -10.40% 14736.0 18438.0 -3702.0 -20.1% 631.0 241.0 390.0 161.8% 6.0 3.0 3.0 100.0%
12 S3b 5 years 3121 2250 871 38.7% 11.254 12.552 -1.298 -10.30% 13430.0 16251.0 -2821.0 -17.4% 629.0 206.0 423.0 205.3% 7.0 5.0 2.0 40.0%
13 S3c 3 years 3099 2256 843 37.4% 11.241 12.550 -1.309 -10.40% 14727.0 23972.0 -9245.0 -38.6% 422.0 114.0 308.0 270.2% 10.0 9.0 1.0 11.1%
14 S3c 5 years 3036 2224 812 36.5% 11.248 12.549 -1.301 -10.40% 12968.0 20278.0 -7310.0 -36.0% 416.0 107.0 309.0 288.8% 11.0 10.0 1.0 10.0%
15 S3d 3 years 3117 2263 854 37.7% 11.239 12.549 -1.310 -10.40% 19458.0 28245.0 -8787.0 -31.1% 304.0 86.0 218.0 253.5% 13.0 11.0 2.0 18.2%
16 S3d 5 years 3045 2227 818 36.7% 11.232 12.548 -1.316 -10.50% -38068.0 27591.0 -65659.0 -238.0% -359.0 62.0 -421.0 -679.0% 16.0 13.0 3.0 23.1%

This didn’t have a big impact though, so it appears the issue may be elsewhere.

13.13-13.25, 14.24-14.41, 14.58-16.00: Comparing against their markdown file and finding right columns

The original repository included a markdown file from a run of this code. Instead of comparing against the paper, I tried comparing against those results, as that should definitely match up, and can tell us if this code as it is is running as expected or not.

I copied the sall tables from Case_Detection_Results.md and Case_Detection_Results_5yrs.md into a single .csv file, then add that to the comparison.

Comparing the paper and repo against my results:

pd.read_csv('sall_compare_1e6_to_mine.csv')
Scenario Interval CostpAgent CostpAgent_paper CostpAgent_repo CostPaperDiff CostRepoDiff CostPaperPerc CostRepoPerc QALYpAgent QALYpAgent_paper QALYpAgent_repo QALYPaperDiff QALYRepoDiff QALYPaperPerc QALYRepoPerc ICER ICER_paper ICER_repo ICERPaperDiff ICERRepoDiff ICERPaperPerc ICERRepoPerc IncrementalNMB IncrementalNMB_paper IncrementalNMB_repo INMBPaperDiff INMBRepoDiff INMBPaperPerc INMBRepoPerc
0 S1a 3 years 2430 2438 2439 -8 -9 -0.3% -0.4% 12.558 12.560 12.560 -0.002 -0.002 0.00% 0.0% 13791 19632 18767 -5841 -4976 -29.8% -26.5% 726 444 482 282 244 63.5% 50.6%
1 S1a 5 years 2355 2356 2355 -1 0 0.0% 0.0% 12.560 12.556 12.557 0.004 0.003 0.00% 0.0% 9740 19847 17603 -10107 -7863 -50.9% -44.7% 845 312 375 533 470 170.8% 125.3%
2 S1b 3 years 2365 2363 2365 2 0 0.1% 0.0% 12.550 12.554 12.552 -0.004 -0.002 0.00% 0.0% 17035 25894 28168 -8859 -11133 -34.2% -39.5% 409 198 167 211 242 106.6% 144.9%
3 S1b 5 years 2311 2296 2298 15 13 0.7% 0.6% 12.554 12.552 12.551 0.002 0.003 0.00% 0.0% 10380 23187 29091 -12807 -18711 -55.2% -64.3% 614 168 105 446 509 265.5% 484.8%
4 S1c 3 years 2398 2386 2386 12 12 0.5% 0.5% 12.552 12.551 12.551 0.001 0.001 0.00% 0.0% 17438 46956 37782 -29518 -20344 -62.9% -53.8% 456 15 76 441 380 2 940.0% 500.0%
5 S1c 5 years 2300 2313 2311 -13 -11 -0.6% -0.5% 12.544 12.550 12.551 -0.006 -0.007 0.00% -0.1% 27839 38673 34119 -10834 -6280 -28.0% -18.4% 119 47 74 72 45 153.2% 60.8%
6 S2a 3 years 2877 2286 2895 591 -18 25.9% -0.6% 12.292 12.553 12.282 -0.261 0.010 -2.10% 0.1% 7853 18908 21840 -11055 -13987 -58.5% -64.0% 1191 223 298 968 893 434.1% 299.7%
7 S2a 5 years 2813 2246 2825 567 -12 25.2% -0.4% 12.287 12.551 12.278 -0.264 0.009 -2.10% 0.1% 9106 17514 19840 -8408 -10734 -48.0% -54.1% 642 176 240 466 402 264.8% 167.5%
8 S3a 3 years 3074 2234 3071 840 3 37.6% 0.1% 11.248 12.548 11.237 -1.300 0.011 -10.40% 0.1% 7930 30366 34146 -22436 -26216 -73.9% -76.8% 803 54 82 749 721 1 387.0% 879.3%
9 S3a 5 years 3031 2207 3024 824 7 37.3% 0.2% 11.235 12.548 11.236 -1.313 -0.001 -10.50% 0.0% -80780 22636 22117 -103416 -102897 -456.9% -465.2% -228 68 152 -296 -380 -435.3% -250.0%
10 S3b 3 years 3187 2292 3195 895 -8 39.0% -0.3% 11.247 12.553 11.247 -1.306 0.000 -10.40% 0.0% 14736 18438 19575 -3702 -4839 -20.1% -24.7% 631 241 466 390 165 161.8% 35.4%
11 S3b 5 years 3121 2250 3113 871 8 38.7% 0.3% 11.254 12.552 11.243 -1.298 0.011 -10.30% 0.1% 13430 16251 17541 -2821 -4111 -17.4% -23.4% 629 206 388 423 241 205.3% 62.1%
12 S3c 3 years 3099 2256 3121 843 -22 37.4% -0.7% 11.241 12.550 11.242 -1.309 -0.001 -10.40% 0.0% 14727 23972 21507 -9245 -6780 -38.6% -31.5% 422 114 299 308 123 270.2% 41.1%
13 S3c 5 years 3036 2224 3056 812 -20 36.5% -0.7% 11.248 12.549 11.240 -1.301 0.008 -10.40% 0.1% 12968 20278 17870 -7310 -4902 -36.0% -27.4% 416 107 274 309 142 288.8% 51.8%
14 S3d 3 years 3117 2263 3132 854 -15 37.7% -0.5% 11.239 12.549 11.243 -1.310 -0.004 -10.40% 0.0% 19458 28245 21157 -8787 -1699 -31.1% -8.0% 304 86 322 218 -18 253.5% -5.6%
15 S3d 5 years 3045 2227 3066 818 -21 36.7% -0.7% 11.232 12.548 11.237 -1.316 -0.005 -10.50% 0.0% -38068 27591 25402 -65659 -63470 -238.0% -249.9% -359 62 158 -421 -517 -679.0% -327.2%

Comparing my results and the repo against the papers results:

pd.read_csv('sall_compare_1e6_to_paper.csv')
Scenario Interval CostpAgent_paper CostpAgent CostpAgent_repo CostMyDiff CostMyPerc CostRepoDiff CostRepoPerc QALYpAgent_paper QALYpAgent QALYpAgent_repo QALYMyDiff QALYMyPerc QALYRepoDiff QALYRepoPerc ICER_paper ICER ICER_repo ICERMyDiff ICERMyPerc ICERRepoDiff ICERRepoPerc IncrementalNMB_paper IncrementalNMB IncrementalNMB_repo INMBMyDiff INMBMyPerc INMBRepoDiff INMBRepoPerc
0 S1a 3 years 2438 2430 2439 8 0.30% -1 0.00% 12.560 12.558 12.560 0.002 0.00% 0.000 0.00% 19632 13791 18767 5841 42.4% 865 4.6% 444 726 482 -282 -38.8% -38 -7.9%
1 S1a 5 years 2356 2355 2355 1 0.00% 1 0.00% 12.556 12.560 12.557 -0.004 0.00% -0.001 0.00% 19847 9740 17603 10107 103.8% 2244 12.7% 312 845 375 -533 -63.1% -63 -16.8%
2 S1b 3 years 2363 2365 2365 -2 -0.10% -2 -0.10% 12.554 12.550 12.552 0.004 0.00% 0.002 0.00% 25894 17035 28168 8859 52.0% -2274 -8.1% 198 409 167 -211 -51.6% 31 18.6%
3 S1b 5 years 2296 2311 2298 -15 -0.60% -2 -0.10% 12.552 12.554 12.551 -0.002 0.00% 0.001 0.00% 23187 10380 29091 12807 123.4% -5904 -20.3% 168 614 105 -446 -72.6% 63 60.0%
4 S1c 3 years 2386 2398 2386 -12 -0.50% 0 0.00% 12.551 12.552 12.551 -0.001 0.00% 0.000 0.00% 46956 17438 37782 29518 169.3% 9174 24.3% 15 456 76 -441 -96.7% -61 -80.3%
5 S1c 5 years 2313 2300 2311 13 0.60% 2 0.10% 12.550 12.544 12.551 0.006 0.00% -0.001 0.00% 38673 27839 34119 10834 38.9% 4554 13.3% 47 119 74 -72 -60.5% -27 -36.5%
6 S2a 3 years 2286 2877 2895 -591 -20.50% -609 -21.00% 12.553 12.292 12.282 0.261 2.10% 0.271 2.20% 18908 7853 21840 11055 140.8% -2932 -13.4% 223 1191 298 -968 -81.3% -75 -25.2%
7 S2a 5 years 2246 2813 2825 -567 -20.20% -579 -20.50% 12.551 12.287 12.278 0.264 2.10% 0.273 2.20% 17514 9106 19840 8408 92.3% -2326 -11.7% 176 642 240 -466 -72.6% -64 -26.7%
8 S3a 3 years 2234 3074 3071 -840 -27.30% -837 -27.30% 12.548 11.248 11.237 1.300 11.60% 1.311 11.70% 30366 7930 34146 22436 282.9% -3780 -11.1% 54 803 82 -749 -93.3% -28 -34.1%
9 S3a 5 years 2207 3031 3024 -824 -27.20% -817 -27.00% 12.548 11.235 11.236 1.313 11.70% 1.312 11.70% 22636 -80780 22117 103416 -128.0% 519 2.3% 68 -228 152 296 -129.8% -84 -55.3%
10 S3b 3 years 2292 3187 3195 -895 -28.10% -903 -28.30% 12.553 11.247 11.247 1.306 11.60% 1.306 11.60% 18438 14736 19575 3702 25.1% -1137 -5.8% 241 631 466 -390 -61.8% -225 -48.3%
11 S3b 5 years 2250 3121 3113 -871 -27.90% -863 -27.70% 12.552 11.254 11.243 1.298 11.50% 1.309 11.60% 16251 13430 17541 2821 21.0% -1290 -7.4% 206 629 388 -423 -67.2% -182 -46.9%
12 S3c 3 years 2256 3099 3121 -843 -27.20% -865 -27.70% 12.550 11.241 11.242 1.309 11.60% 1.308 11.60% 23972 14727 21507 9245 62.8% 2465 11.5% 114 422 299 -308 -73.0% -185 -61.9%
13 S3c 5 years 2224 3036 3056 -812 -26.70% -832 -27.20% 12.549 11.248 11.240 1.301 11.60% 1.309 11.60% 20278 12968 17870 7310 56.4% 2408 13.5% 107 416 274 -309 -74.3% -167 -60.9%
14 S3d 3 years 2263 3117 3132 -854 -27.40% -869 -27.70% 12.549 11.239 11.243 1.310 11.70% 1.306 11.60% 28245 19458 21157 8787 45.2% 7088 33.5% 86 304 322 -218 -71.7% -236 -73.3%
15 S3d 5 years 2227 3045 3066 -818 -26.90% -839 -27.40% 12.548 11.232 11.237 1.316 11.70% 1.311 11.70% 27591 -38068 25402 65659 -172.5% 2189 8.6% 62 -359 158 421 -117.3% -96 -60.8%

I initially misunderstood this, and started troubleshooting, revisiting the installation instructions:

devtools::install_github('KateJohnson/epicR', ref="closed_cohort")

And how this fails because rredis has been removed from CRAN - hence, the method I had used instead, of loading the local folder as a package.

ERROR: dependency ‘rredis’ is not available for package ‘epicR’
* removing ‘/home/amy/Documents/stars/stars-reproduce-johnson-2021/reproduction/renv/library/linux-ubuntu-jammy/R-4.4/x86_64-pc-linux-gnu/epicR’
Warning message:
In i.p(...) :
  installation of package ‘/tmp/RtmpuwIUmF/file723a23834b02/epicR_1.27.7.tar.gz’ had non-zero exit status
Reflection

Worth being aware of this for an R package, that dependencies becoming unavailable can make this installation method impossible.

However, I then realised my mistake, and revisiting the tables, recognised that actually:

  • For costs and QALYS, my results and the respository results were very similar (and both were very different from the paper for scenarios 2 and 3)
  • For ICER and INMB, my results were very different to the repository for all scenarios, which were in term very different to the paper (i.e. none match!)

This gives me assurance that I am getting the right results for costs and QALYs, but that likely there is a difference I hadn’t noticed.

I then wondered if perhaps I were looking at the wrong table from the .md file.

Reflection

Having instructions in repository to guide me to the tables, or having labels in the .Rmd itself to say this was Table 3 and so on, would have been very helpful here, as it seems to be taking me a while to get my head around!

The Cost Effectiveness Plane table is “adjusted to the total population”. I made a .csv with the CEPlane from their .md and then modified Process_Model_Results.Rmd to use CEplane instead of sall.

In that table, you can see CostpAgentAll comes out lower than CostpAgent, for example, which matches up to paper, so I used those “All” columns.

This resolved the issue for costs and QALYS, which now both matched the paper:

pd.read_csv('tab3_compare_to_original.csv')
Scenario Interval CostpAgent_paper CostpAgentAll CostpAgentAll_repo CostMyDiff CostMyPerc CostRepoDiff CostRepoPerc QALYpAgent_paper QALYpAgentAll QALYpAgentAll_repo QALYMyDiff QALYMyPerc QALYRepoDiff QALYRepoPerc ICER_paper ICER ICER_repo ICERMyDiff ICERMyPerc ICERRepoDiff ICERRepoPerc IncrementalNMB_paper INMB INMB_repo INMBMyDiff INMBMyPerc INMBRepoDiff INMBRepoPerc
0 S1a 3 years 2438 2430 2439 8 0.3% -1 0.0% 12.560 12.558 12.560 0.002 0.0% 0.000 0% 19632 13791 18767 5841 42.4% 865 4.6% 444 703 446 -259 -36.8% -2 -0.40%
1 S1a 5 years 2356 2355 2355 1 0.0% 1 0.0% 12.556 12.560 12.557 -0.004 0.0% -0.001 0% 19847 9740 17603 10107 103.8% 2244 12.7% 312 868 411 -556 -64.1% -99 -24.10%
2 S1b 3 years 2363 2365 2365 -2 -0.1% -2 -0.1% 12.554 12.550 12.552 0.004 0.0% 0.002 0% 25894 17035 28168 8859 52.0% -2274 -8.1% 198 386 131 -188 -48.7% 67 51.10%
3 S1b 5 years 2296 2311 2298 -15 -0.6% -2 -0.1% 12.552 12.554 12.551 -0.002 0.0% 0.001 0% 23187 10380 29091 12807 123.4% -5904 -20.3% 168 637 142 -469 -73.6% 26 18.30%
4 S1c 3 years 2386 2398 2386 -12 -0.5% 0 0.0% 12.551 12.552 12.551 -0.001 0.0% 0.000 0% 46956 17438 37782 29518 169.3% 9174 24.3% 15 433 40 -418 -96.5% -25 -62.50%
5 S1c 5 years 2313 2300 2311 13 0.6% 2 0.1% 12.550 12.544 12.551 0.006 0.0% -0.001 0% 38673 27839 34119 10834 38.9% 4554 13.3% 47 142 110 -95 -66.9% -63 -57.30%
6 S2a 3 years 2286 2284 2287 2 0.1% -1 0.0% 12.553 12.555 12.552 -0.002 0.0% 0.001 0% 18908 7853 21840 11055 140.8% -2932 -13.4% 223 703 186 -480 -68.3% 37 19.90%
7 S2a 5 years 2246 2235 2245 11 0.5% 1 0.0% 12.551 12.548 12.550 0.003 0.0% 0.001 0% 17514 9106 19840 8408 92.3% -2326 -11.7% 176 418 137 -242 -57.9% 39 28.50%
8 S3a 3 years 2234 2220 2232 14 0.6% 2 0.1% 12.548 12.551 12.548 -0.003 0.0% 0.000 0% 30366 7930 34146 22436 282.9% -3780 -11.1% 54 556 45 -502 -90.3% 9 20.00%
9 S3a 5 years 2207 2218 2207 -11 -0.5% 0 0.0% 12.548 12.537 12.548 0.011 0.1% 0.000 0% 22636 -80780 22117 103416 -128.0% 519 2.3% 68 -126 67 194 -154.0% 1 1.50%
10 S3b 3 years 2292 2273 2289 19 0.8% 3 0.1% 12.553 12.550 12.553 0.003 0.0% 0.000 0% 18438 14736 19575 3702 25.1% -1137 -5.8% 241 440 243 -199 -45.2% -2 -0.80%
11 S3b 5 years 2250 2258 2248 -8 -0.4% 2 0.1% 12.552 12.548 12.551 0.004 0.0% 0.001 0% 16251 13430 17541 2821 21.0% -1290 -7.4% 206 366 183 -160 -43.7% 23 12.60%
12 S3c 3 years 2256 2233 2255 23 1.0% 1 0.0% 12.550 12.545 12.550 0.005 0.0% 0.000 0% 23972 14727 21507 9245 62.8% 2465 11.5% 114 246 148 -132 -53.7% -34 -23.00%
13 S3c 5 years 2224 2219 2221 5 0.2% 3 0.1% 12.549 12.545 12.549 0.004 0.0% 0.000 0% 20278 12968 17870 7310 56.4% 2408 13.5% 107 251 113 -144 -57.4% -6 -5.30%
14 S3d 3 years 2263 2241 2261 22 1.0% 2 0.1% 12.549 12.545 12.550 0.004 0.0% -0.001 0% 28245 19458 21157 8787 45.2% 7088 33.5% 86 230 124 -144 -62.6% -38 -30.60%
15 S3d 5 years 2227 2222 2226 5 0.2% 1 0.0% 12.548 12.539 12.548 0.009 0.1% 0.000 0% 27591 -38068 25402 65659 -172.5% 2189 8.6% 62 -42 67 104 -247.6% -5 -7.50%

This aligns with the table 3 caption from paper about symptomatic patients (scenario 2) and smoking history (scenario 3):

“The ‘All patients’ strategy encompassed the entire population of interest. 59% of the population was included in the ‘Symptomatic’ strategy, and 46% in the ‘Smoking history’ strategy. To maintain a constant reference population, per patient costs and QALYs were adjusted to include patients not selected by the strategy. For the symptomatic strategy, the costs (and QALYs) shown are the sum of the costs (QALYs) of patients not included in the symptomatic strategy, and the costs (QALYs) of those included, weighted by the proportion in each group. The results of smoking history strategy were adjusted in the same fashion” Johnson et al. (2021)

16.02-16.33: Looking at ICER and INMB

I tried to see if I was using the wrong ICER, or what might explain the difference.

Looking at table 3 in the paper, if I were to calculate the ICER manually from change in costs over change in QALYs, I would get:

(2438-2151)/(12.560-12.546)
20499.999999998356

20499 is quite different from 19632, although I am assuming that might be because I am oversimplifying in this calculation, and that other things went into that calculation.

Looking at my results, I wondered if I might be using the wrong column, as we have ICER and ICERAdj. I had a look at the original output of incrementalcosts and incrementalqaly in my .md file and the original, and realised that actually although the costs and qalys are very similar, it has a big impact on the ICER. Therefore, it seems this could be related to the number of agents.

Timings

import sys
sys.path.append('../')
from timings import calculate_times

# Minutes used prior to today
used_to_date = 371

# Times from today
times = [
    ('09.28', '09.38'),
    ('10.10', '10.19'),
    ('10.20', '11.30'),
    ('11.32', '11.36'),
    ('13.08', '13.10'),
    ('13.13', '13.25'),
    ('14.24', '14.41'),
    ('14.58', '16.00'),
    ('16.02', '16.33')]

calculate_times(used_to_date, times)
Time spent today: 217m, or 3h 37m
Total used to date: 588m, or 9h 48m
Time remaining: 1812m, or 30h 12m
Used 24.5% of 40 hours max

References

Johnson, Kate M., Mohsen Sadatsafavi, Amin Adibi, Larry Lynd, Mark Harrison, Hamid Tavakoli, Don D. Sin, and Stirling Bryan. 2021. “Cost Effectiveness of Case Detection Strategies for the Early Detection of COPD.” Applied Health Economics and Health Policy 19 (2): 203–15. https://doi.org/10.1007/s40258-020-00616-2.