ECMWF's Next-Generation IO and Product Generation for the IFS Model

Starting 2014, ECMWF has embarked on a 10 year research programme on HPC Scalability, aiming to achieve Exascale numerical weather prediction systems by 2025. ECMWF operational forecast generates massive amounts of I/O in short bursts, accumulating to tens of TiB in hourly windows. From this output, millions of user-defined daily products are generated and disseminated to member states and commercial clients all over the world.

These products are processed from the raw output of the IFS model, within the time critical path and under strict delivery schedule. Upcoming rise in resolution and growing popularity will increase both the size and number of these products. Based on expected model resolution upgrades, by 2020 we estimate the operational model will output over 100 TiB/day and need to archive over 400 TiB/day. Given that the I/O workload is already one of the strongest bottlenecks in ECMWF's workflow, this is one of the main challenges to reach Exascale NWP.

We present the latest ECMWF developments in model I/O and product generation, and how we are reworking our operational workflows to adapt to forthcoming new architectures and memory-storage hierarchies. In particular, we also present a new interpolation package and associated the new product generation system, which we plan to bring into operations in 2018.