Reviewing code and running model with reproduction of results between runs. Total time used: 4h 36m (11.5%)
Looked over code and data from original study, taking notes below.
Tried using version 4, found it worked really well except:
Hence, I’m going to suggest that I don’t think it is worth trying to use this tool, and that a simpler and more standardised approach would be visual comparison of figures, or overlaying of the figures when possible.
All items are figures, so no actions required.
Capacity_Model_Template.ipynb creates figures incredibly similar to the article. Can spot a few slight differences likely due to different parameters. Some examples: * Figure 2, slightly wider confidence intervals in the notebook * Figure 4 number of displaced patients, different “bumps” in the line
From article:
Within environment.yaml, can see packages and versions, to create a conda environment.
Copied environment.yaml file into reproduction. Run the command conda env create --name covid19 --file environment.yaml from terminal within reproduction folder to create environment.
Found that:
This was then successful, and built quickly (within 30s). A quick glance over the environment confirmed that it looked to have the correct python version and packages.
conda env update --file environment.yaml --prune. This fixed the error.Protocol suggestion: On reflection, perhaps it would’ve better to copy over the whole code folder into reproduction/ and then modify there, rather than copying over stuff bit by bit?
Switched to doing that -
Images look similar to their notebook from Zenodo but, as with those figures, slight differences to paper, likely due to parameters.
But can’t decide how best to display it - or if it is worth trying to display it all, or better just to suggest to browse the code repository themselves. Feels like an unhelpful distraction at present, and misleading compared to just allowing people to browse the code, so have suggested to change it to just displaying the PDFs and call it a day.
Updated the files accordingly.
Original (Allen et al. (2020)):
Run 1:
Run 2:
number_of_replications = 30
scenarios = {}
scenarios['base_3_month'] = Scenario(
run_length=150,
total_proportion_people_infected = 0.8,
time_to_infection = Normal(60, 15, 0.0),
time_positive = Uniform(7, 14),
proportion_pos_requiring_inpatient= 0.6,
time_pos_before_inpatient = Uniform(3,7),
time_inpatient = Uniform(7.0, 14.0),
mortality = 0.15,
random_positive_rate_at_start = 0.0
)Thinking differences are defintely down to this. Thinking options are:
GitLab history:
Looked into GitLab history at https://git.exeter.ac.uk/tmwm201/dialysis-service-delivery-covid19/-/commits/master.
Can see there are releases more recent than Zenodo (22 April), which I had used, and the paper (received 28 april 2020, published 13 Aug 2020). All commits since the Zenodo publication was:
None of these would have any impact on results (besides it being re-run). Does confirm correct version of Joblib was used.
Then looked back over older releases, but these were only just prior to the Zenodo publication (first was 20 April) and none use random state etc.
Run more replications than 30
Although it doesn’t match with the paper, I tried increasing replications to see what the figure averaged out to. However, then concluded that wasn’t particularly helpful in trying to get their results (although could be a way of saying that these are “more stable” results - although, it would impact on the confidence intervals and so on - if not able to set a seed).
Modifying script to set a random seed, to make the result I am getting from this code reproducible
Change random_state in parameters.py from None to having a value for each in their class:
Original (Allen et al. (2020)):
Run 1 with seeds added:
Run 2 with seeds:
However, this definitely has not fixed the issue! Still varying - and I wouldn’t be able to reproduce my own results twice either.
At the moment, I would describe this model as reusable but not reproducible. It was really relatively quick to get the code up and running and see similar results to the paper. But in terms of getting it to match up to the paper, it is pretty much impossible, although I will try to get their via setting random seeds then running it lots of times to try and get a close match.
This is important for STARS framework improvement - that controlling randomness is important for reproducibility. It can also be handy for someone reusing a model, as they may wish to reproduce just to verify that its running properly for them.
And so for each of the studies, if this is a recurring thing that comes up, its seeing where and how to add random seeds in different models and languages, to enable reproducibility.
From this Stack Overflow post, I’m suspicious that perhaps the issue is that I am setting the random state as 1 and 2, which (a) would imply it’s making it the same between each run, but (b) all using the same stream in parallel processing. But it’s set using RandomState.
Trying to google around use of seeds with parallel processing.
Had a chat with Tom about it and he suggested:
requiring_inpatient_random: Distribution = Uniform(0.0, 1.0)time_pos_before_inpatient: Distribution = Uniform(3,7)Need seperate random number streams for each time make a distribution to use it.
I thought best option is to switch to using it how it is uses in the treat-sim model docs, as focus here is just modifying code to allow it to reproduce each run.
So next things I did -
Then ran it twice again (this time just with 5 replications). Not matching up yet -
Run 1 with new random method:
Run 2:
import sys
sys.path.append('../')
from timings import calculate_times
# Minutes used prior to today
used_to_date = 73
# Times from today
times = [
('10.24', '11.00'),
('12.10', '12.16'),
('12.19', '12.29'),
('13.26', '13.52'),
('14.19', '15.12'),
('15.23', '16.35')]
calculate_times(used_to_date, times)Time spent today: 203m, or 3h 23m
Total used to date: 276m, or 4h 36m
Time remaining: 2124m, or 35h 24m
Used 11.5% of 40 hours max✅ = Made the change.
💬 = Noted to discuss with team.
Protocol:
Change section of reproducing items to:
Suggestions to consider for research compendium stage: