ECMWF's Next Generation IO for the IFS Model

Starting 2014, ECMWF has embarked on a 10 year research programme on HPC Scalability, aiming to achieve Exascale numerical weather prediction systems by 2025. ECMWF operational forecast generates massive amounts of I/O over very short bursts, accumulating to tens of TB/day, from which millions of user-defined products are generated. Based on expected model resolution increases, by 2020 we estimate the operational model will output hundreds of TB/day and need to archive over a PB/day.

Given that the I/O workload is already one of the strongest bottlenecks in ECMWF's workflow, this is one of the main challenges to reach Exascale NWP. Following the recent award of EU funding with the NextGenIO H2020 project, ECMWF is researching how to best apply forthcoming I/O technologies to its workflow, in particular to the interface between the IFS model output and the product generation.

The project partners include hardware vendors with whom we aim to (1) co-design an HPC system integrating forthcoming Non-Volatile RAM (NVRAM) technologies and develop appropriate API's in support of hierarchical storage systems; (2) investigate the optimal way to incorporate NVRAM technologies into an HPC workload, in particular ECMWF's operational workflow (3) develop a I/O software layer that delivers IFS model output to product generation without using the parallel I/O system (Lustre/GPFS) but retaining the resilience characteristics needed by an operational workflow.