8 Modifying the framework

8.1 Contents of the framework

Section from reflections	Consideration for framework
List required packages	The difference between listing all packages, and listing some that then have other dependencies. The various options there are for listing packages, and the downside of just assuming its obvious from imports. The options for guiding dependency management (e.g. providing scripts that download packages, and then environment management tools)
Provide version for Python/R and for packages	Important to do so for both (not just packages). Challenge of implementing in R. Ease of implementing in Python. Importance of listing in repository itself. Options for doing this in R and Python. Suggestions. Need to explore more with R. Is it a matter of providing renv but suggesting someone could try with latest versions in first instance? As not realistic to guarantee maintenance of research projects. But know at least when it worked - as being clear, it won’t always, even in R
Model in a “runnable” format	This was an unusual case, but I think could merge more under the example of providing code that aligns with the paper, as in that case, it was the code for a web app when the paper wasn’t about that
Model designed to run programmatically, and don’t hard code parameters you want to change.	Consider whether framework will given guidance on model structure. If so, it should be suggesting that (A) parameters are seperate from model and (B) run scenarios by changing elsewhere (and not by directly modifying the model code). Why do this? Because its simpler to run multiple versions of model with same script, and reduces likelihood of missing errors of inputing wrong parameters
Avoid large amounts of code duplciation	This would also be a code structure thing (does it fit in framework?). Good to do in general - fairly standard coding practice. Here as it makes code more readable, makes it easier when want to change all scenarios, and reduces likelihood of introducing mistakes (which did see)
Sufficient comments in code	Likewise, code structure. Standard practice like docstrings (which only one or two had). The standard structures that are available for docstrings (e.g. roxygen in R). And then also just general comments in code itself.
Quicker models	Worth considering for framework, as this was practically one of the big things for me when it came to using the models. Think about the options that are available for reducing model run time. And from the start, principles of keeping it simple and small. Things to avoid that make it slow. Alternatives to those things. The value of a quicker model (e.g. in being able to make an app, to rerun it all easily and spot mistakes, to just be easier to work with).
State run time	Could include in minimum documentation
State memory usage and alterantives for lower spec machines	Could include in enhanced documentation? Or minimum? Need to consider that this can seem a daunting thing if not familiar and wouldn’t know how to find this out or what this means
Provide code for all scenarios
Include correct parameters in script	Suggestion of default model parameters matching up to baseline in paper, and explaining that if not, issue becomes that its hard to check if model is all correct without code or paper listing every single parameter
Provide all required parameters
Clearly present parameters in paper	This would not be relevant for STARS framework, but could feed into STRESS-DES work? How it being in a table was so much easier than it being described in the text, and how it being comprehensive of all parameters that get varied (or clear if those are the only things changed from baseline, and so on)
Provide calculations	Again, might not quite fit into STARS? But if you are going to be mentioning the “pre-processed” values at all, then its important to include the calculation (ideally in the code, as that is the clearest demonstration of exactly what you did)
Saves output to a file	Code structure type thing. This was really handy, particularly for long run times. In doing this, set up in way that is easy to change name and location of output file, and not hard coded to one thing
Understandable output tables	Not sure where/if this might fit. But suggestion is to not provide alternative results for same metrics. And if a table or output might be unclear on what need or how to calculate, then providing some docs or something that supports
Using seeds	Explain how, how it can be quite simple, and what benefit of this is. Could be in enhanced
Provide code to produce (a) tables (b) figures (c) in-text results	Explaining why this is handy (being able to reproduce gives confidence model is running right and also important in science) (plus useful for self too if need to make changes in review process and so on). Noting importance of not forgetting about “in-text” results
Instructions on how to run model	Already in minimal documentation, but be aware, even as simple as “run this script and here is an example of how” is helpful
Grid lines	Minor, not sure if fits/to include
Data dictionaries	Minor and intersects with good commenting, might not be relevant for all, but also worth considering, if people are uploading data as part of model, what the principles and practices and recommendations are for shairng of data and linking to those (as this is likely part of those, along with other things). If there is overlap, make it clear the synergies (e.g. if talks about archiving data, how could do alongside code, and so on).

8.2 Presentation of the framework

Checklist table with categories, description and space to complete. But in multiple synced formats (perhaps auto conversion github action between them) - e.g. markdown, latex, docx

Diagram: Think about any ways could adapt this and if would benefit from that. E.g. gradual reveal with GIF. more icons. alt layout. etc.

Website: Could share like https://joss.readthedocs.io/en/latest/paper.html but would want to make sure we are adding sufficient detail on top of checklist.

Interactive web app: Could make a web app that serves as a “form” for people to complete and then downloaded their completed framework like there is set up for the PRISMA checklist.

8.3 Reflections from the evaluations

TBC - in progress below…

Although we can comment on how commonly or uncommonly something met a criteria, it is only a small sample (n=6), so when covered in the DES best practice audit, that might be more informative, as baesd on far more studies.

From evaluation of repository:

Open license - available on about half, added on request for rest, a very important one as without it you technically cannot reuse someones code
Complete materials - rarely, so alot of time spent on writing code and figuring out how best to do that
Dependency management - rarely done, and although not an issue in simpler cases, a big impact in more complex
Model interface - provided for two although not helpful as (1) host no longer operating, and as not focus, didn’t try to run locally, and (2) other app was viewable but didn’t produce the figures and outcomes from paper (was more general, whilst paper delved into scenarios beyond app)

From evaluation of paper:

It often took a long time (up to 2 hours). It would be valuable if the completed checklist were provided along with the paper, like how people often do with equator network reporting guidelines like PRISMA. Consider varying levels detail its possible to complete (yes/no, linking to sections, full detail). Linking to sections is minimum needed to be really useful. Yes/no doesn’t impact time to find information really.
Input parameters - very handy, as if uncertain in code, use this to check or code
Some of my evaluations were more light handed (e.g. STRESS 1.2 and 1.3) whilst some were more nit-picky
Run length - important for knowing when its slow
Scenarios - important to be clear, felt provided in a table was the clearest

Reflections on STRESS-DES from having worked on it:

1.2 “how and when calculated” might feel like excess detail
1.3A providing name for each scenario doesn’t always seem necessary
2.4 initially found confusing/irrelevant (e.g. “operation of conveyor”)
3.2 would be helpful to have more examples
5.1 listing all packages and their versions is never particularly achievable or feasible. More feasible is specifying main package used for model (e.g. simmer, simpy, base R, numpy/pandas) and providing finer details in code (although it was helpful to have version of this package in the paper, when no versions were mentioned in the GitHub)
5.2 Algorithm used is tricky/uncommon knowledge
5.3 Event-processing machanism is tricky/uncommon knowledge
Too many different things in several (could benefit from being clearer, seperated, or into components like 2.5). This was the case for 1.3, 3.3, 4.1, 4.3, 5.1, 5.2, 5.3, 5.4

Reflections from comparing what is in STRESS-DES v.s. ISPOR-derived:

Scenarios - ISPOR has a simpler description
Model uncertainities - nice that ISPOR requires this
ISPOR has lots of items that relate to validity and how “good” it is (9 to 18) - whilst STRESS doesn’t touch on this