Using Python in the Cloud to Generate Hydrometeorological Data Products in Real-Time from Terabytes of Geospatial Data

DASSP is a scalable, cloud-based platform for real-time high-resolution global ground state forecasting that uses python in multiple contexts. DASSP predicts high-resolution soil moisture, soil strength, and water/energy fluxes at 1-arc-second resolution in the Continental U.S. (CONUS) and 3-arc-second resolution globally. Applications such as disaster management, agriculture, water resources management, and military mobility prediction benefit from high-resolution ground state predictions. DASSP predicts soil moisture using a physics-based model to disaggregate low-resolution weather-driven land surface model (LSM) predictions. This model combines information from higher resolution data-sources to distribute the low-resolution soil moisture prediction in a mass-conserving fashion. In this talk, we will describe the different ways that python is used to support the DASSP platform. We will also touch on how open source development practices and version control using git was used to integrate team members with various backgrounds and levels of experience. Python is an excellent tool for integrating all of the geospatial data sources, algorithms, and analyses required by a system as complex as DASSP. We use simple python scripts to download, collect, stage, and batch-processes the various data sources, automating otherwise tedious manual tasks. We also use python for computationally expensive analyses. The python package pyDEM was developed to precompute the topographic wetness index (TWI), which is the slowest part of the prediction algorithm. pyDEM takes a directory of elevation data files and computes the aspect, slope, upstream contributing area, and TWI, in parallel, and without edge effects. This package has been used to calculate the TWI for CONUS at 1 arc-second resolution, which consists of over 3,500 files, each containing 3612x3612 arrays of gridded data, or 45 gigapixels. At the heart of the DASSP platform is an object-oriented python package that combines datasources and algorithms in a pipeline architecture. The pipeline architecture allows data from various sources having various resolutions and coordinates frames to be displayed, or combined in a calculation. DASSP can combine the low-resolution land surface model predictions with high-resolution TWI data, landcover data, and soil properties. It can also be used to probe point measurements from the International Soil Moisture Network (ISMN) or area measurements from the AIRMOSS project. Additionally, we use ipython notebooks for internal examples and data analysis. Within the notebook, the core DASSP software is used as a library to analyze data and predictions for algorithm development, scientific exploration, and validation studies. The notebook is an excellent tool for recording and refining these analyses. Python is an integral tool in all of these different use cases. We also heavily leverage open source python packages. In particular, to serve our products via a webconsole, we use a heavily customized version of MetaCarta's TileCache server. We also use GDAL, geopy, pyproj, and shapely for geospatial data management and access. Numpy, scipy, and cython are used for computational efficiency, and pandas is used for working with labelled, and messy datasources (where random values are missing). These packages greatly increase the development of products within DASSP, and the quality of various open source libraries has been outstanding. We have also benefitted from open source development practices using git and github to collaborate effectively within a team of developers. Likewise, we have released pyDEM as open source to provide a useful library to the wider open source community.