The European Centre for Medium-Range Weather Forecasts Approach to Cloud Computing

As most meteorological organisations, the European Centre for Medium-Range Weather Forecasts (ECMWF) is facing the challenges created by ever growing data volumes. The Centre needs to make the results of its forecasting systems available to its Member States, Cooperating States in a timely fashion. This is currently done by pushing the data to users via a private network or the Internet, according to a prescribed schedule; this system is limited by the disk and network bandwidths. As a consequence, users cannot make use of the vast number of products generated, in particular the output of the Centre’s ensemble forecasting systems. As part of the Centre’s strategy, it is expected that the horizontal resolution of ensemble forecasts will reach 5km by 2025, making the issue of dissemination and uptake further acute. ECMWF is also running one of the largest meteorological archives in the world, currently holding over 250 PB of data, and growing at a rate in excess of 1 PB per week. This archive is expected to reach 1 EB in the second half of 2020 and 2 EB by the end of 2023. Thousands of users are issuing millions of requests per day to retrieve data from popular datasets such as ERA5, the Centre’s fifth generation reanalysis produced in the context of the European Union’s Copernicus programme. Once again, usage of such data is limited by slow downloads over the Internet.

ECMWF believes that cloud-based services offer a way forward the address these challenges and embracing cloud technologies is now part of its strategy.

ECMWF plans to offer to its Member States and Cooperating States the ability to create virtual machines in a cloud infrastructure that is located in its own data centre, close to its high-performance computing facility and its data handling system. Although the virtual machines would be hosted on the Centre’s premises, its Member States and Cooperating States will have full control over them and could consider offering services to their own end-users, for example by running a web site offering products derived from ECMWF model outputs. Partnership with commercial cloud providers will also be investigated for providing elasticity solutions for users that have needs that exceed what the baseline infrastructure can provide. Finally, it is foreseen that this infrastructure will be federated with other ones from certain Member States and EUMETSAT and will serve the entire European Meteorological Infrastructure.

For this project, three challenges have been identified: technological, legal and security. These are currently being studied, and ECMWF is testing a series of uses cases with some of its users.