Bridging HPC and Data Analytics for NWP - ECMWF Present and Future

ECMWF's Scalability programme is now entering its 2nd phase, where focus is now on improvements to the operational forecasting system that incur deeper restructuring changes. The adoption of a new object store (FDB version 5) for the time-critical operations has opened the door for more comprehensive improvements to the post-processing chain. These improvements will bring product generation and data analytics closer to the NWP model and the model output data, to build true data-centric processing and analytics workflows.

These are part of ECMWF plans to achieve Exascale NWP by 2025 and to empower our users and member states with novel and increased usage of our weather forecast data.
As Exascale NWP datasets are expected to feature between 250 TiB to 1 PiB per forecast cycle, the data-centric approach is critical to enable their efficient usage, by minimising data transport and bringing post-processing and insight discovery closer to the data source.

We will present multiple projects that ECMWF is embarking on to achieve this vision. These include:
(1) Refactoring the existing I/O server to fully support all model components (FVM, ocean, chemistry, etc.).
(2) Introduce support for interpolation and product-generation on-the-fly and in memory, together with user-defined processing pipelines.
(3) Provide object store API's for fast access to the data hypercubes across axes that previously were very costly or inefficient (time-series, multi-ensemble, vertical profiles).
(4) Enable high-performance data analytics (HPDA) and machine learning (ML) algorithms on top of those data hypercubes for time-critical real-time forecasts.
(5) Create a real-time notification system to allow early data usage by end-users at the source.

Part of the above work is co-funded by the European Union, under project LEXIS with Grant Agreement 825532.