ASSISTT Regression Testing: Quality Control of Scientific Algorithm Source Code Updates

The Algorithm Working Group (AWG) Algorithm Scientific Software Integration and System Transition Team (ASSISTT) maintains and updates a data processing framework for the GOES-R data processing algorithms. This system provides an environment for algorithm development and testing along with the ability to process multiple algorithms in sequence with product precedence. The ASSISTT maintains a standard test data set which is used for routine regression testing of the AWG algorithms. As modifications to the framework data processing system are made over time due to new algorithms, new input data for current algorithms, or other software updates and fixes, it is important that the system as a whole remains stable in terms of the algorithm output products, i.e. changes in one part of the Framework do not inadvertently affect other parts. In particular, software updates to an algorithm should not alter another non-related algorithm's outputs. Routine regression testing is undertaken by the ASSISTT to ensure that the Framework system is stable in these regards by comparing algorithm output products before and after updates to the Framework are made. Guaranteeing the integrity of the algorithms is an important part of the ASSISTT's role in quality assurance for algorithms being transitioned from research to operations.

The standard test data set for regression testing is chosen so that algorithm coverage (i.e. the number of lines of source code) is maximized during algorithm processing. Ensuring algorithm code coverage via selection of the test data is a crucial part of the quality control process. If only partial algorithm coverage is achieved during testing, it is not possible to determine if a particular code modification has impacted the regression results. In order to optimize the selection of the test data, the standard GNU utility ‘Gcov' is used to determine which test data maximizes algorithm coverage. Test data that maximizes coverage for each AWG algorithm is chosen for the standard test data set with the goal of overlapping input data for multiple algorithms in order to minimize storage of test data and save computer resources during algorithm processing. This presentation describes the methods that were used to create the standard test data set for the AWG algorithms, shows the test cases for several algorithms, and describes the use of Jenkins as our routine testing tool.