Lossless differential compression of weather radar data in universal format using motion estimation and compensation

A meteorological radar data assimilation system has been developed at the Marine Meteorology Division of the Naval Research Laboratory to enhance the safety of ship and aircraft operations.  Radar observations are assimilated into the Navy's Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS^®) to improve the 0-24 hour forecasts of hazardous weather and to provide decision makers with timely products to help exploit or mitigate those predictions.  The system takes advantage of Navy vessels having weather processors for their tactical radars (e.g., SPS-48E/G: Hazardous Weather Detection and Display Capability; SPY-1 Tactical Environmental Processor).  The ships in the battle fleet having this capability will be able to digitally generate full-resolution, full-volume weather radar data, and archive those data in Universal Format (UF) files approximately every 5 minutes.  Several UF files will be transmitted in near-real-time per hour to Fleet Numerical Meteorology and Oceanography Center where the data assimilation into COAMPS^® is conducted.  UF file sizes range from ~5 MB (SPS-48E) to ~13 MB (SPY-1), which would be too large a load on the operational bandwidth of the ships' communication systems.  However, the raw weather radar data have large dynamic ranges and turned out to be difficult to compress losslessly. This problem was tackled by applying the following methods sequentially: (i) A slightly lossy method of scaling down the data by a factor of ten to their meteorological values, followed by rounding to the nearest integers; (ii) A totally lossy approach of aggressive data reduction via thresholding operations on the scaled-down data. The thresholds were so chosen that the new data after the thresholding operation would still maintain sufficient information content to impact analysis and forecast with COAMPS^®; and (iii) A intra-UF file compression software package based on the open-source bzip2 algorithm, which achieves significant lossless compression on the UF file headers and the thresholded data, by exploiting spatial correlations and header redundancies. Although significant reduction on the UF file sizes could be achieved through the above methods, strictly lossless compression of the original raw data has not been directly addressed.

In this work we considered the differential (delta) compression technique, which is concerned with efficient file transfer over a bandwidth-constrained link in the case where the receiver already has similar file (or files). For example, the so-called Delta Compressor zdelta has been used to increase the efficiency of distributing updated versions of software over a network, or synchronizing personal files between different accounts and devices. Specifically, we study the feasibility of increasing the lossless compression efficiency on the original data, by developing inter-UF file compression algorithms that can exploit the temporal correlations between successive UF files that were taken approximately 5 minutes apart. We analyzed archived SPS-48E UF data obtained from an at-sea experiment onboard the USS PELELIU (LHA5) in February 2006. This data set contains a wide range of precipitation echoes spanning 22 hours of observations. Simulations showed that neither differential compression based on straightforward UF file subtractions, nor the use of Delta Compressor zdelta would lead to more efficient compression than the existing intra-UF file compression technique.  To overcome this challenge, we employ motion estimation techniques to both capture the temporal correlations between successive UF files and to compensate the motions of the ship and the weather prior to differential coding. Strictly lossless coding is accomplished by compressing the motion parameters and the residue signals after motion compensation using entropy codes.

Lossless compression experiments on the USS PELELIU data set demonstrated that our differential compression technique based on motion estimation and compensation was effective in capturing inter-UF file correlations, thereby leading to about 4% on average more efficient compression than the existing intra-UF file compression technique, if the latter is applied on the original raw data (i.e., without data scaling, rounding and thresholding).  Furthermore, compared with our previously developed intra-coding technique for headers, about 40% more compression on average could be attained by differential compression of the headers in successive UF files, where headers account for a very significant portion (about 30%) of the overall UF file size.  

It should be noted that the our new inter-UF file compression method is completely lossless and more than meets the 1 MB maximum file size requirement imposed by the operational bandwidth. However, in terms of overall file size reduction, it cannot compete with the existing intra-UF data reduction and compression technique, which achieves ~40% on average more reduction in file sizes, when data unscaling, rounding and thresholding are included.

To the best of our knowledge, this is the first effort to apply motion estimation and compensation methods employed by lossy video compression standards (such as MPEG and H.26x) to addressing the problem of lossless compression of weather radar data. Further improvement of the method might be possible by refining the techniques for motion estimation and compression of motion parameters. While providing a useful lossless data compression option for the current efficient file transfer problem, our new differential compression technique may also find beneficial applications in efficient data archiving applications, where strictly lossless data compression is often a requirement.

COAMPS^® is a registered trademark of the Naval Research Laboratory