Standardizing Observation Data for the A2e WFIP2 Project

The Atmosphere to Electrons (A2e) Data Archive and Portal (DAP) provides the critical service of a data repository for all data produced by DOE EERE’s A2e program. This includes a wide variety of observational and model datasets that are expected to be 1PB within the year. However, the DAP seeks to do more than preserve the data for access by researchers but also make it useable. In order for the important work of inter-comparison and use by modelers the observation data sets need to be in a consistent format that is readily useable. Observation data sets are often loosely structured in text or other formats which requires a data user to parse the syntax of each file type. For the Wind Forecast Improvement Project 2 (WFIP2), there were more than 90 observation datasets that covered the 18 months of the experiment. Utilizing software libraries developed for the DOE BER ARM program, a suite of processes were developed to produce standardized data products for all WFIP2 observation data. The focus was on producing consistent netcdf files that captured the primary measurements of each dataset. Doing this work has enabled the further work of the WFIP2 project to focus their efforts on using the data instead of parsing it, and also lays a foundation for standardizing additional data products to benefit future data users of A2e data. This poster presents the architecture and information on the state of this system as it functions in the DAP.