ARM Data: Metrics, Processing, and Metadata Management for Evaluation Data Products

Atmospheric Radiation Measurement (ARM), a U.S. Department of Energy (DOE) scientific user facility, is a key geophysical data source for national and international climate research. The ARM Data Center (ADC) processes over 1.7 petabytes of stored data, including over 16 TB of data originating from instruments or computational algorithms that are in early stages of development. These “evaluation” datasets are made available for scientists to use in a “beta testing” mode to provide feedback for continual improvement of the quality and scope of the dataset. In an effort to streamline the delivery of evaluation data to the scientific community, we have implemented ARM standard naming conventions, file formatting, and metadata for optimized accessibility through the ARM Data Discovery Portal. Data will be easier for the user community to utilize and evaluate, as well as facilitate future transition to routine ARM data products covering a wider spatial and temporal scale. This presentation will first provide an overview of the purpose and overall impact of ARM data. We will then detail the steps taken to transition legacy evaluation data from our outdated Intensive Operation Period (IOP) area to our vast data archive where routine ARM data products are stored in a standardized directory structure. Python and SQL scripting were used to preserve existing metadata and create complete metadata records that meet rigorous ARM metadata standards. This project will detail an updated workflow to process new evaluation data products and highlight products available to the scientific user community.