LIVVkit 2: A Robust and Extensible Python Package for Ice-sheet Model Verification and Validation

To confidently make real world predictions with an ice-sheet model, such as sea-level rise or ice-shelf breakups, scientists require the model to have a high level of credibility. Verification and validation (V&V) encompasses a set of techniques which quantify confidence and can be used to effectively build model credibility within the scientific community. To meet this need, there is an ongoing effort to develop LIVVkit, the land ice verification and validation toolkit, within the Community Ice Sheet Model (CISM) development community. LIVVkit is a python based robust and extensible verification and validation package which covers both software V&V (the construction and use of the model) and model V&V (the mathematics and physics within the model).  

Intrinsically, V&V is a continuous process such that each model change requires a new round of V&V testing. LIVVkit easily integrates into the development cycle, allowing developers to quickly verify their changes, integrators to robustly variety large sets of changes, and modelers to run a variety of validation tests. Within the ice-sheet modeling community, many different ice-sheet models are under active development, new validation datasets are becoming available, and new methods of analysing these models are actively being researched. In order to keep pace with the scientific and model developments, LIVVkit includes a model V&V framework that can easily be extended to incorporate new tests.  

Python has, in many ways, been essential to the development of LIVVkit. Importantly, the python standard library includes a wide variety of tools needed to develop a V&V package. Overall, LIVVkit uses 21 modules from the standard library to: take in configurable use input, determine the desired set of V&V analyses need to be run, pull in any custom user analyses, execute the full set of analyses in parallel, and generate a portable hierarchical website describing the results. In fact, LIVVkit only requires four python packages not included in the standard library: numpy, scipy, matplotlib, and netCDF4-python. These packages allow LIVVkit to load in ice-sheet model data, perform all the complex mathematical analyses, and then generate a set of high quality figures. Fortunately, all of these are available through python's standard package managers like Anaconda or PIP, which significantly reduces the complexity of deploying LIVVkit on the large variety of machines and operating systems used throughout the ice-sheet modeling community.  

Furthermore, LIVVkit is intended to not only be used by scientists, but extended by them as well. Thus, usability and familiarity are essential design elements of LIVVkit. Due to python's rich set of scientific packages (e.g., numpy, scipy, pandas), many ice-sheet scientists are already familiar with python and routinely use python for data analysis, making it easy for them to understand and extend LIVVkit.  Python's standard library module Argparse is a robust command line interface module which allows us to provide an interface that mimics the way scientists typically interact with ice-sheet models. Because of this widespread adoption, python and its scientific packages are available on high-performance computing systems like Titan at the Oak Ridge Leadership Computing Facility. This is essential to LIVVkit's ability to evaluate the large datasets coming out of fully coupled earth system models (which include ice-sheet models).  

Finally, LIVVkit is expanding beyond the ice-sheet modeling community. By using modern python packaging techniques and isolating any ice-sheet specific code into model 'bundles', the core functionality of LIVVkit can be distributed and adopted into the larger testing harnesses in use by the Earth system models or used to analyze an entirely different component (LIVVkit may function as a V&V library). Currently, there is active research for the atmosphere component of the Accelerated Climate Model for Energy (ACME) devoted to determining whether different model ensembles contain the 'same' climate. To support this research, LIVVkit is developing an atmosphere bundle to extend coverage to the atmostphere model. This will allow rapid adoption and comparison of ensemble testing in step with the research and development efforts.

Our presentation will demonstrate a V&V analysis using LIVVkit, show how to create a custom user analysis, and highlight how python made this all possible.