Computation, Analysis and Visualization of In-Situ and Remote Sensing Data using Python

The demand for weather, water, and climate information has been high, with an expectation of long, serially complete observational records spanning from minute to century timescales. The completeness of these data is essential to the assessment of Earth's climate in reports such as the Bulletin of the American Meteorological Society's (BAMS) State of the Climate and the US Global Change Research Program's (USGCRP) National Climate Assessment. Over the next few years, the amount of in-situ and remote sensing data ingested by the National Oceanic and Atmospheric Administration's (NOAA) National Climatic Data Center (NCDC) will reach the order of petabytes. In addition, there is a need for openness and transparency, to ensure the integrity and reproducibility of data from original observation to final product.

Scientists at the Cooperative Institute for Climate and Satellites (CICS), located within NCDC, have begun to address these concerns using the Python open source language. Here we will present advancements and challenges of utilizing climate data through packages such as NumPy, SciPy, MPI, and matplotlib. Projects include time series analysis of surface temperature observations, visualization of weather and climate data to meet sector needs, storage of large datasets into user-friendly formats such as CF compliant netCDF, and traceability through metadata tracking. Performance will be compared with conventional languages to demonstrate Python's speed and efficiency. We hope these practices will create a center-wide awareness of utilizing the power of Python in NCDC processing.