Events Data Base for Data Retrieval from the CASA DFW X-band Radar Network

Improving the Accessibility and Usability of Data the CASA DFW X-Band Radar Network

Aleksei Rutkovskii, Kathleen McLaughlin, David Westbrook, Brenda Philips, and Michael Zink

Paros Center for Atmospheric Research

University of Massachusetts

Amherst, MA 01003

In atmospheric sciences, data availability and usability are essential for advancing mesoscale, synoptic, and climate research and applications. Atmospheric data is gathered through various methods, including satellite observations, ground-based stations, radar systems, and emerging technologies like drones and IoT sensors. Each method contributes unique data; however, there are many challenges associated with accessing different atmospheric science datasets. The data is often archived based on time and location rather than specific meteorological phenomena, making it difficult for researchers to access data without prior knowledge of the phenomena (Oakley & Daudert 2016). Additionally, the size and diversity of atmospheric data, collected at varying temporal and spatial resolutions, require knowledge of data formats such as NetCDF, which require specialized software and expertise to interpret, as well as substantial computational resources for storage, processing, and analysis, often necessitating sophisticated data management techniques (Veillette et al. 2020). For example, working with ERA5, a comprehensive reanalysis dataset used to develop and train machine learning models, involves managing vast amounts of data across multiple variables like temperature, humidity, and wind speed. This requires large computational resources for downloading, storage, and preprocessing. This workflow is complex, challenging for beginners, and often slow due to the need for specialized knowledge and tools to handle large datasets and perform the necessary data transformations. Moreover, with the increase in machine learning research, more researchers are facing these types of workflows. Thus, the complexity presents a significant barrier to broad data accessibility and usability, slowing down research progress and innovation in atmospheric science.

This research addresses how to make CASA X-band Radar data more broadly available for atmospheric research. The CASA DFW Network is a network of 4-7 X-band radars in North Texas that have been generating high spatiotemporal resolution single radar moments and multi-radar products since 2016 (Philips et al. 2017). This research is inspired by SEVIR, a curated and annotated collection of over 10,000 weather events. SEVIR is specifically designed for deep learning applications in meteorology, providing temporally and spatially aligned image sequences depicting weather events captured over the contiguous US (CONUS) by the GOES-16 satellite and the mosaic of NEXRAD radars (Veillette et al. 2020). SEVIR offers a streamlined, user-friendly approach, allowing scientists to focus more on model development and analysis rather than on data preparation. The curated nature of SEVIR ensures that it is immediately useful for specific machine learning applications.

In our work, we extend the idea of data availability presented by SEVIR by adding a web interface that allows users to navigate and selectively extract atmospheric data of interest. For the CASA data, we enable researchers to use the NOAA Storm Events database and CASA’s own events archive to easily discover meaningful weather events for atmospheric research at the scale relevant to CASA data. A key advantage of our platform is its dual approach to data access: the web interface allows users to visually explore and select specific data subsets of interest, while access via the Zarr format is facilitated by reference files generated using Kerchunk (https://fsspec.github.io/kerchunk/), enables efficient multiprocess reading directly from the cloud. This dual access method combines the simplicity of web-based data exploration with the power of scalable, programmatic data processing, accommodating a wide range of research needs. To support this enhanced accessibility and usability, the data is stored in a cloud S3 bucket hosted on the Open Storage Network (https://www.openstoragenetwork.org/), ensuring high availability and reliability. The backend and frontend of the platform are hosted on NERC (https://nerc.mghpcc.org/) or secure web providers such as Amazon Web Services or Google Cloud Platform. By providing this state-of-the-art solution that enhances both data availability and usability, we aim to significantly advance research in mesoscale meteorology, numerical weather prediction (NWP), machine learning (ML), and climate science. Our platform enables researchers to more rapidly and effectively develop predictive models and applications, leveraging both the accessibility of the web interface and the efficiency of Zarr-based data access for high-performance computing environments.

References

Oakley, N. S., and B. Daudert, 2016: Establishing best practices to improve usefulness and usability of web interfaces providing atmospheric data. Bull. Amer. Meteor. Soc., 97, 263–274, https://doi.org/10.1175/BAMS-D-14-00121.1.

Philips, B., and Coauthors, 2017: Tracking tornados down streets: Using CASA radars in real-time severe weather warning operations in North Central Texas. IEEE Int. Geosci. Remote Sens. Symp., 2017, 5973–5976, https://doi.org/10.1109/IGARSS.2017.8128370.

Veillette M., and Coauthors, 2020: SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite meteorology. Adv. Neural Inf. Process. Syst., 33, 22009–22019.