Using Python as a Tool for Unifying Data Collection, Storage, and Visualization

The Python language provides functionality that makes it possible to easily implement basic data parsing, such as reading data from a URL, converting the data into Python objects, and saving the data in a text file. While the ability to perform this simple operation is a keystone computational operation of the NM Climate Center office, it is also necessary to be able to collect data from our in-situ sensor assets, store the data reliably, perform more advanced processing on the data, and format the data for office/public consumption. To implement the collection, storage, processing, and formatting of data in multiple languages would be a time intensive task and it would complicate the maintenance of the resulting disparate code-base. Fortunately, the Python community provides numerous packages that extend the functionality of the base language, making it possible for Python to be used as a “glue” between computational resources. For example, an in-situ weather station could be connected to a small computer, such as a Raspberry Pi, running Python packages that can be extended to automatically process and transfer data to our servers (weewx, PyCampbellCR1000). The XML data from third-party sources may be parsed using the BeautifulSoup Python package. Python interfaces for popular databases such as MySQL and PostgreSQL allow for long term data storage and retrieval. Array based data processing are provided by the well known NumPy package. Python based web frameworks, such as Django, tie the storage and processing of data together with the powerful data formatting options so that data can be made available on the web. Furthermore, popular open-source desktop applications, such as QGIS, have built-in Python bindings, allowing our office to design scripts that connect to our database and web assets to easily generate PNG map images or GIS shapefiles.

Because of the capabilities provided by Python and external packages written by the active development community, it is an excellent platform to use in building the software systems that collect, store, process and display, environmental data for the state of New Mexico. It is the purpose of this paper to describe the use of Python at the NM Climate Center in support of our data operations and showcase several time-series and spatial products made possible by using Python and available third-party packages.