Fast, Easy, Open: Sustaining Live Updating, Analysis Ready, Cloud Optimized Weather Data Archives

Accessing weather data at scale is a challenge. Fast, easy, cheap: most solutions meet only one or two of those criteria. Integrating each new data source often requires writing bespoke code and the resulting download speeds constrain large scale analysis and AI model training. These challenges limit the pace and ambition of weather and climate research and innovation – challenges the authors have experienced directly and seen collaborators encounter in settings ranging from academic research to startups to large weather impacted organizations.

We present dynamical.org, an open data effort to create and sustain analysis-ready, cloud optimized (ARCO) weather datasets. From a technical standpoint, we share how dynamical.org live updates datasets, while maintaining the internal data organization (“chunks“) that enable low latency and high throughput access for a wide range of usage patterns. From an organizational standpoint, we share how the legal structure, governance, and financial model is designed for long-term continuity and operational reliability. Finally, we share real-world examples of how these datasets are accelerating the deployment of weather dependent decision making systems.