Exploring Using Artificial Intelligence (AI) for NWP and Situational Awareness Applications

The volume and diversity of environmental data obtained from a variety of Earth-observing systems, has experienced a significant increase in the last couple years with the advent of high spectral, high- spatial and temporal resolutions sensors. At the same time, users-driven requirements, especially for nowcasting and short-term forecasting applications but also for medium-range weather forecasting, strongly point to the need for providing this data in a consistent, comprehensive and consolidated fashion, combining space-based, air-based and surface-based sources, but at higher spatial and temporal resolutions and with low latency. This trend is expected to continue further with the emergence of commercial space-based data from multiple industry players and the advent of flotillas of small satellites (Cubesats) as well as new sources of data (such as Internet of Things IoT) to complement traditional environmental data. Yet, the data volume presents already a significant challenge. Satellite measurements input to data assimilation algorithms for instance, need to be aggressively thinned spatially, spectrally and temporally in order to allow the products generation, calibration, assimilation and forecast system. Only a fraction of satellite data gets actually assimilated. Taking full advantage of all the observations, allowing more sources of observations to be used for initial conditions setting, and to do it within an ever shrinking window of assimilation/dissemination, requires exploring new approaches for processing the data, from ingest to dissemination. We present in this study the results of a pilot project’s effort to use cognitive learning approaches for numerical weather prediction (NWP) applications. The Google’s machine learning open-source tool TensorFlow, used for many Artificial Intelligence (AI) applications, was used to reproduce the performances of remote sensing and data assimilation techniques to fuse data from many sources including satellites, with flexibility to extend to other sources such as IoT. The outcome is a 5D-cubeset of parameters to describe the state of the Environment, useful for multiple applications including NWP. The approach relies on training a deep-layer neural network on a set of inputs from NASA’s GEOS-5 Nature Run (NR) and corresponding observations simulated based on it using the Community Radiative Transfer Model (CRTM) and other forward operators. The present study demonstrates the proof of concept and shows that using AI holds significant promise in potentially addressing the vexing issue of computational power and time requirements needed to handle the extraordinary volume of environmental data, current and expected. With dramatically lower execution times, we will compare the AI-based algorithm performances to those of a variational algorithm used to perform data assimilation pre-processing and retrievals, as well as compare the AI-based assimilation system to those of an ensemble/variational hybrid data assimilation system used in NOAA operations (GSI) and explore assessing its potential impact on NWP applications. Comparisons will be based on the NR set up and therefore the truth will be known exactly.