A new method for compressing quality-controlled weather radar data by exploiting blankout markers due to thresholding operations

This work addresses the feasibility of further improving the efficiency of compressing a weather radar data set, as reported in the paper titled “Efficient reduction and compression of weather radar data in universal format”, appearing in the Proc. of AMS 25th Conference on International Interactive Information and Processing Systems (IIPS'09) for Meteorology, Oceanography, and Hydrology, Phoenix, AZ, January 2009.

As determined by some appropriate quality control requirements on weather radar applications, thresholding operations could be employed to blank out data entries whose values are below certain thresholds and thus considered less critical. For example, as shown in the IIPS'09 paper, if a raw reflectivity (DZ) data entry has a value below 5 dBZ, then this data entry would be blanked out and the corresponding data location would be labeled with a marker with a certain value. As a result of such a thresholding operation, over 40% of the DZ data entries were found to be blanked out and thus their locations were labeled with markers with identical values. While the data compression method presented in the IIPS'09 paper achieved a high efficiency, it did not directly exploit these markers that comprise a significantly large proportion of the data to be compressed.

This work investigates a new data compression method, which divides the thresholded data entries into two categories: blankout markers and critical data entries, and compresses these two types of data entries separately. While markers were introduced by the thresholding operation to replace less critical data, their locations convey important spatial information that must be retained. To this end, a lossless approach is proposed to compress the spatial information and perfectly reconstruct the information. On the other hand, with the markers in the original thresholded data being skipped, the critical data entries can be accumulated and reorganized into a new chunk of data, which tend to have smaller entropy than the original thresholded data, thereby making higher compression on the data possible. Preliminary results showed that this new method could provide further compression on the original thresholded data than the existing method. More detailed discussions on the method and simulation results will be given in the extended abstract.