Monday, 7 January 2019: 2:30 PM
North 123 (Phoenix Convention Center - West and North Buildings)
In the past five years, ECMWF has invested significant resources in the enhancement of efficiency and scalability of the forecasting system - from observational data handling at the beginning of the processing chain and data assimilation to running the forecast model and dealing with vast amounts of model output data at the end of the chain. Despite these efforts it is estimated that there will be substantial shortcomings in computing power for reaching ECMWF’s ambitious goal of running a global Earth-system forecast ensemble at 5 km spatial resolution in 2025 while meeting the same time constraints as today.
To address key structural bottlenecks in prediction workflows, ECMWF’s current focus is on more radical ways to enhance efficiency along two main lines of research: the separation of concerns in programming and a fully integrated and distributed model output data handling.
The separation of concerns aims to create flexibility for algorithmic and numerical method choices at the science-code level and highly specialised modules that exploit various processor technology options at the kernel level. In between, options for domain-specific languages supported by generic libraries for performing operations on model fields and optimizing memory access and parallelism are being developed. This effort is strongly supported by European-Commission funded projects like ESCAPE (hpc-escape.eu), ESCAPE-2 and EuroEXA (euroexa.eu), together with many European partners.
The data handling strategy aims to overcome the file oriented and disk-based approach and integrates data operations into the pre and post-processing chain through object stores much closer to where data is needed or created. This offers options to exploit new memory layers with less latency and high access bandwidth during model runtime, and flexible work management with distributed processing for model output post-processing. This work is based on similar partnership projects like NextGenIO (nextgenio.eu) and MAESTRO.
The talk will show progress in these research projects but will also touch on the bigger picture of weather and climate prediction in Europe, that is currently proposed as the European Flagship project ExtremeEarth (extremeearth.eu).
To address key structural bottlenecks in prediction workflows, ECMWF’s current focus is on more radical ways to enhance efficiency along two main lines of research: the separation of concerns in programming and a fully integrated and distributed model output data handling.
The separation of concerns aims to create flexibility for algorithmic and numerical method choices at the science-code level and highly specialised modules that exploit various processor technology options at the kernel level. In between, options for domain-specific languages supported by generic libraries for performing operations on model fields and optimizing memory access and parallelism are being developed. This effort is strongly supported by European-Commission funded projects like ESCAPE (hpc-escape.eu), ESCAPE-2 and EuroEXA (euroexa.eu), together with many European partners.
The data handling strategy aims to overcome the file oriented and disk-based approach and integrates data operations into the pre and post-processing chain through object stores much closer to where data is needed or created. This offers options to exploit new memory layers with less latency and high access bandwidth during model runtime, and flexible work management with distributed processing for model output post-processing. This work is based on similar partnership projects like NextGenIO (nextgenio.eu) and MAESTRO.
The talk will show progress in these research projects but will also touch on the bigger picture of weather and climate prediction in Europe, that is currently proposed as the European Flagship project ExtremeEarth (extremeearth.eu).
- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner