Leveraging FAIR Principles for Efficient Management of Meteorological Radar Data

Radars play a critical role in meteorology due to their high spatio-temporal measurement resolution, enabling early detection and tracking of severe weather phenomena. These capabilities empower meteorologists to issue timely alerts and warnings, thereby safeguarding lives and reducing property damage. Beyond immediate surveillance and forecasting needs, radar data serves many offline applications, including statistical analyses of clouds and precipitation, climatological studies, and data analysis for insurance risk assessment, among others. These products rely on analyzing radar data as a time series, underscoring the versatility and utility of radar information in long-term meteorological studies. A radar volume scan, comprising data collected through multiple cone-like sweeps at various elevation angles, often exceeds several megabytes in size every 5 to 10 minutes that is usually stored as individual files. Consequently, national weather radar networks accumulate vast amounts of data with non-interconnected files over extended periods spanning several decades. This presents significant challenges in data storage and availability, particularly when treating radar data as a time-series dataset, which parallels the complexities of managing big data.

Traditionally, radar data storage involves proprietary formats that demand extensive input-output (I/O) operations, leading to prolonged computation times and high hardware requirements. In response, our study introduces a novel data model designed to address these challenges. Leveraging the Climate and Forecast (CF) format-based FM301 hierarchical tree structure, endorsed by the World Meteorological Organization (WMO), and Analysis-Ready Cloud-Optimized (ARCO) formats, we developed an open data model to arrange, manage, and store radar data in cloud-storage buckets efficiently. This approach uses a suite of Python libraries, including Xarray (Xarray-Datatree), Xradar, Wradlib, and Zarr, to implement a hierarchical tree-like data model. This model is designed to align with the new open data paradigm, emphasizing the FAIR principles (Findable, Accessible, Interoperable, Reusable). Focusing on the Carimagua, Colombian radar data, we demonstrate the efficacy of querying, selecting, and computing monthly rainfall accumulation in radar coordinates using our proposed hierarchical tree-like data model stored in ARCO format. Our results reveal significantly faster data processing times than legacy data models on standard hardware configurations, such as an Intel Core i7 processor, 32 GB DDR4 RAM, and an NVME solid-state disk drive laptop. Furthermore, FAIR open radar data represents a promising solution for managing large, live-updating radar datasets within cloud platforms. By adhering to the open science principles, this approach makes scientific data more accessible across all levels of society.