Data Compression with Deep Neural Network and Principal Component Analysis Machine Learning

Given the need of transferring a very large output dataset from numerical weather prediction (NWP) model in low bandwidth environments, it is still a challenge to compress 4D data without losing its original spatial variability. The Naval Research Laboratory (NRL) developed a machine learning-based approach to efficiently compress and decompress a modified refractivity output dataset from COAMPS (Coupled Ocean/Atmosphere Mesoscale Prediction System). This model was built for a duo purpose quickly compress and decompress variable-sized data and reserve spatial variability of its original. The framework implements a two-step model system: Deep Neural Network (DNN) model and Principal Component Analysis (PCA). A Deep Neural Network (DNN) model capable of compressing modified refractivity output from COAMPS areas into a one-dimensional array. Then, Principal Component Analysis (PCA) is utilized as a preliminary step, ensuring timely results, and later refines the output through a second model based on historical performance metrics from PCA. A quantitative assessment and evaluation based on an independent data set show promising results. With a compression ratio of 1983:1 and mean squared error (MSE) of 0.0259. The overall compressed file size is reduced to 47 KiB, while compared to the original size of approximately 91 Mib. This machine learning-based compression approach presents a viable solution to efficiently compression ratios and minimize data transmission delays. The trade-off of an increase in the loss of spatial variability within original data, as compression ratio increases due to reduced M gradient strength. An overview of frameworks and comparative assessments of results will be presented.