Archiving and Visualizing Ceilometer Data Using Python

A NSF-supported field study of persistent cold-air pools in the Salt Lake Valley, Utah during the 2010-2011 winter led to analyses of complex aerosol layer structures from laser ceilometer observations . The utility of the ceilometers for analyzing boundary layer structures motivated the acquisition and deployment of additional ceilometers for research and real-time monitoring applications, which in turn required developing Python-based software to archive, display, and analyze the resulting data. The PyTables library has been employed with compressed HDF-5 files to develop an archive system designed for both speed and disk space efficiency. The structure of the file system we have developed will be discussed, along with techniques to query datasets and preserve disk space. Leaflet map-tile software has been used to efficiently display real-time and archived time-height datasets, which allows complex visualizations with dynamic properties, such as variable plotting ranges and color bars. The techniques used to present this information, including memory caching, PIL, SocketServers and Tornado servers will be discussed.