OSSEs in the cloud: Tomorrow.io’s Observation System Simulation Experiment system on Azure

The past decade has witnessed a revolution in weather data. SmallSat and CubeSat constellations are now feasible and can be rapidly prototyped, built, and launched. These constellations offer greatly-enhanced global coverage and greatly-reduced data latencies compared to standalone observing platforms. Tomorrow.io will launch two satellite series in 2024 featuring microwave radars and radiometers with the goal of improving precipitation nowcasts and forecasts.

We have established a series of Observation System Simulation Experiments (OSSEs) to validate these constellations before launch to demonstrate their impact on a forecast through both improved data volume (e.g., more satellites) and improved data availability (e.g., rapid refresh and reduced observation latencies). We use the Joint Effort for Data assimilation Integration (JEDI) project to construct a framework to run these OSSEs, as JEDI’s modular design is flexible and well-suited for next-generation experiments. We describe how we generate our simulated microwave observations using our RadSim code, and we also describe how our observation generation and experiment cycling systems drive JEDI and are used to rapidly launch studies for both scientific research and for system profiling.

A major challenge in deploying JEDI is to maximize performance on a computational platform with heterogeneous compute resources, that is with differing compute node resources and with different MPI topologies. These platforms are increasingly common following a shift toward cloud computing, and cloud resources may be tailored to specific applications. We use Microsoft Azure’s CycleCloud product to simulate these environments and derive performance and tuning statistics for JEDI, and we compare results against bare-metal HPC setups.