Lessons Learned Processing Data from Live GOES-16 Data Using the Operational Algorithms

This presentation highlights the experiences and lessons learned while integrating live GOES-16 data from the operational Product Distribution (PD) Two Day Store into an end-to-end test-bed for assessing operational Level 2 algorithm performance. Part of these activities were spent tailoring infrastructure software and systems to the GOES-16 data. As a result, much was learned about managing and processing the data and valuable insights were revealed, resulting in better, more effective algorithm execution.

A key task during this process was tuning of the many parameters governing algorithm execution. Key elements of this process are: (a) close cooperation between software and science personnel; (b) careful and unified configuration management of the parameters and software; and (c) an integrated set of support tools. All of this has led to the development of tools and techniques to more efficiently process the data.

There are several key skill sets are also required among an integration team: (a) understanding of the data format and organization; (b) data requirements of each component of the system; (c) engineering of applicable tools and infrastructure; and (d) data analysis and validation.

AER used the Algorithm Workbench (AWB) as its primary integration tool. AWB is a highly scalable light weight integration suite that was developed around the GOES-R system. AWB can be used in both research and operational environments, and is portable between platforms from a small laptop to a high-powered cluster and even to the cloud. AWB leverages the ease of use of a Python language interface with the performance of customized embedded C++ modules for algorithms and other pixel-level tasks.

Techniques for sharing, organizing and updating data were developed and many best practices were developed. We successfully ran end-to-end execution of algorithms for the complete GOES-16 processing chain. By analyzing the data, we were able to identify certain caveats and adjust system and algorithm parameters to provide better outputs. These results were directly applied to the operational system.

These experiences highlight the continuing need for the use of standard software engineering practices, including common interfaces, code reuse, encapsulation, and portability. The software engineering approach on the GOES-R development program was designed as an effort to generalize the way algorithms are developed and used across research, test and operational environments. By leveraging what we have learned in this experience, future development will become more effective and efficient.