Today's research and operational forecast models and data assimilation systems have difficulty ingesting and utilizing large volumes of satellite data, in part due to prohibitively large computational costs, time constraints and bandwidth issues. To address this problem, NASA recently funded a project aimed at refining, testing and customizing an existing automated Intelligent Data Thinning (IDT) algorithm, developed at the University of Alabama in Huntsville (UAH), in conjunction with commonly used data assimilation systems for numerical weather prediction models. The most significant measure of a successful data reduction algorithm is its ability to retain valuable information – that which has maximum impact on the model forecast – while simultaneously reducing the data volume. The IDT algorithm is specifically designed to retain information-dense regions of a data set while removing redundant data. This recursive simplification algorithm, is based on the computer graphics concept of data decimation, retains data within regions of high spatial frequency (large variances), while subsampling regions of low spatial frequency (low variances) to thin the data.

The goal of this project is to test, refine and customize the existing IDT algorithm in order to transition it into a deliverable data reduction tool useful for real-time applications with a wide variety of dense NASA satellite data streams in operational, research, and private industry communities. The project tasks include: (1) performing sensitivity analysis on IDT with selected data assimilation systems, (2) customizing IDT for use with selected multidimensional NASA satellite data sets, and (3) evaluating IDT's performance with end-to-end analysis and numerical weather prediction model experiments. The NASA Short-term Prediction Research and Transition (SPoRT) Center, with its resident research scientists, forecast models, and real time data feeds, has been chosen as the ideal environment for operational testing of this tool. The refinements made to the IDT algorithm and the results from the sensitivity analysis will be presented in this paper.