Day 10

compendium
guidelines
Author

Amy Heather

Published

July 3, 2024

Note

Finished up research compendium, and discussed uncertain/unmet guidelines with Tom.

Research compendium

  • Docker
    • Add Dockerfile and tested it - opened jupyter lab, ran one of the notebooks, and so confirmed it was working (although took a little longer to run - 5m 50s v.s. 2m 11s for in-text result 5)
    • Activated GitHub action to host on GitHub Container Registry
  • README
    • Corrected and updated

14.00-14.20: Discussion with X about uncertainties and unmet criteria from the evaluation

(Spent roughly half on badges and STARS, and other half on reporting

Badges

Uncertain

Item My comments Discussion
Includes an open license Couldn’t remember if we’d agreed that a license added upon our request should be marked as a condition met or not. Tentatively marked as ‘1’. Not met, but caveat when mention that they add it on request
Complete set of materials shared (as would be needed to fully reproduce article) Uncertain on this, as I had to modify the script alot to get results (i.e. writing scenario logic). However, arguably, I did have all the information I would need to do that? Tentatively marked as ‘1’. Unmet - had to make new materials to reproduce (and not just troubleshooting)
Artefacts are well structured/organised (e.g. to the extent that reuse and repurposing is facilitated, adhering to norms and standards of research community) Does use object-oriented programming (which is great), but parameters are hard coded and uses global variables. Tentatively marked as ‘0’. Agree - good enough for replication but not for reuse (mainly due to hard coding of parameters)

Unmet

All clear cut.

STARS

Uncertain

Item My comments Discussion
Open license. Free and open-source software (FOSS) license (e.g. MIT, GNU Public License (GPL)) As for badges
ORCID. ORCID for each study author Have marked as not met, as there are no ORCID in the repository, although Varun Ramamohan does have ORCID ID with publication in SIMULATION. Agree not met, to be in repositories

Unmet

All clear cut.

Reporting

Uncertain

Item My comments Discussion
STRESS-DES 1.2 Model outputs. Define all quantitative performance measures that are reported, using equations where necessary. Specify how and when they are calculated during the model run along with how any measures of error such as confidence intervals are calculated. Is what I have quoted sufficient for rating of “Partially”? Yes
ISPOR-SDM 12 Is cross validation performed and reported…comparison across similar modeling studies which deal with the same decision problem was undertaken. Is my conclusion reasonable? Fine
ISPOR-SDM 13 Is external validation performed and reported?…the modeler(s) examined how well the model’s results match the empirical data of an actual event modeled. There’s alot of validation work done, but not sure I can spot this, or any justification against it? Change to N/A

Unmet

Change to N/A for 14 as that is about forecasting.

Test-run

Mamba/conda

Tom downloaded the repository to test run the code and see if he could get the same results. He set up a mamba environment and ran pytest, and found he got the same results, and that it took apx. 8 minutes to run on his machine.

Some comments from this exercise were:

  • The bash script source activate shoaib2022 didn’t work if he was in an active conda environment when trying to run the script
    • I was able to replicate this error too
  • The source deactivate command returned an error - although the bash script would continue running fine.
    • I likewise get this error - I presume I might have always been getting that error and previously not noticed
  • It wasn’t immediately clear to him whether it was definitely running or not (as they take a few minutes each), so might be good to add a “what to expect” to the notes
    • On reflection from this conversation, I think it would also be good to include the time expected for pytest
  • My description of pytest was ambiguous - on reading, he had assumed that it required you to have run the bash script first, before running pytest.

Based on the above, I made the following changes to improve the running and clarity of the repository:

  • Removed source deactivate from the bash script
  • Clarified whether to have environments active or not when run command
  • Add detailed “what to expect” from running the bash and pytest options, explaining what should appear on the screen, and how it might then remain blank for a few minutes whilst the model runs
  • Made it clearer that pytest doesn’t depend on bash

Docker

Tom then also tried to use Docker (instead of the conda environment). His reflections from this were:

  • He found that the container built fine.
  • However, before he could do that, he had to get set-up with docker, which now recommends docker-egine.
  • He found that, once the image was activated, he didn’t need to run conda activate shoaib2022 or jupyter-lab - he could simply navigate to the web address to open jupyter lab.
    • I could replicate this - it appears I just never tested it without not running those lines!
  • He wasn’t able to get the image from the GitHub Container Registry - received an authentication error (something to do with personal access tokens)
    • I could also replicate - and I’m suspicious that perhaps I hadn’t double-checked if this had worked prior, and had just assumed it to
  • He also suggested looking for a command that would trigger jupyter lab to open without the user needing to go to that web address
  • He also suggested that, after I’ve made some tweaks from these comments, we could also ask Alison to do a test-run (so - following these changes - it gets checked with a fresh pair of eyes)

Based on the above, I made the following changes:

  • Removed conda activate shoaib2022 and jupyter-lab
  • Add some additional instructions on viewing docker containers and reopening a closed container
  • Changed default docker run command suggestion to be one that will also open the web browser

Went through a few troubleshooting stages for this (which also spilled into 4th July, but included here for clarity):

  • Setting organisation to allow PAT tokens in the organisation settings
  • Running docker login ghcr.io -u githubusername and logging in with a Personal Access Token (classic) (with write:packages and delete:packages), before trying docker pull…
  • Rebuilding the image
  • Running docker logout, and then logging in again, but this time ensuring to use sudo (for both docker login and docker pull - docker login will succeed without sudo, but it seems docker pull required the login to have been with sudo (?)). Opened up jupyter lab using that image and all was fine.