IN016-14
Lessons and challenges in reproducible computational research from the development and application of landscape evolution models to waste site remediation

Wednesday, 9 December 2020: 21:09
Virtual
Katherine R Barnhart, University of Colorado at Boulder, Cooperative Institute for Research in Environmental Sciences, Boulder, CO, United States; USGS Geologic Hazards Science Center, Landslide Hazards Program, Golden, CO, United States, Gregory E Tucker, University of Colorado at Boulder, CIRES and Department of Geological Sciences, Boulder, CO, United States, Sandra G Doty, Consultant, Denver, CO, United States, Rachel Glade, Los Alamos National Laboratory, Geological Sciences, Los Alamos, NM, United States, Charles M. Shobe, Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany; West Virginia University, Department of Geology and Geography, Morgantown, United States, Matthew W Rossi, University of Colorado at Boulder, Earth Lab (CIRES), Boulder, CO, United States and Mary C Hill, University of Kansas, Geology, Lawrence, KS, United States
Abstract:
‘Which model is appropriate?’ is a common question in scientific modeling. We discuss lessons on reproducibility from a set of studies of long-term (10 ky) erosion at a 5 km2 watershed in New York, USA, where environmental waste is stored on-site. Model development, sensitivity analysis, calibration, validation, and prediction under uncertainty took over one million core-hours and produced over 4 TB in output.

We developed 37 alternative models with different governing equations to address this question. The governing equations were established early on, but numerical details were refined throughout the study duration. Ideally, all computation would have been re-done with a final version of the code. However, the required compute-time prevented re-computation. This study exemplifies a common challenge to scientific computing—namely, that applications motivate the creation or modification of software, the construction of scientific software is research in and of itself, and it is done in parallel with implementing applications. These qualities interact with reproduction in several ways. First, a researcher may anticipate the need to re-run simulations themselves, thereby incentivizing easily repeatable workflows. Second, expecting modification of the software motivates use of version control and software testing standards. While this study has not been reproduced, these tools provide us with confidence that while some details of the software may have changed, the results were unaffected.

‘What is the minimum unit to support and assess reproduction?’ is a natural next question. Journals (rightly) require archiving of the ‘data’ to reproduce results. Input scripts and data, source code, and post-processing scripts for this project (64 MB) were within typical archival limits. Yet assessing reproduction requires at least some model output or derivative products, which dramatically increase the data volume. What should be archived? We do not expect a single solution, especially when considering both compute-time and final data volume. For our study, we released all output via the University of Colorado’s PetaLibrary and a non-archival, publicly accessible, read-only Globus Endpoint. Archiving model output is necessary, but the what, where, and for how long remain important, open questions.

<< Previous Abstract | Next Abstract >>