A novel use of Python on a high-performance computer to streamline the forecast production chain

The Met Office Unified Model (MetUM) Gridded Post-Processing System (GPPS) strives to improve the quality of MetUM gridded forecasts and make them useful and relevant to a wide variety of public and commercial customers. The GPPS applies a range of physical and statistical methods to downscale the gridded data from a variety of different MetUM configurations, as well as using the latest surface observations, and radar and satellite imagery, to generate extrapolation-based nowcasts. However, over the years, the GPPS has grown in an organic manner to become a highly complex system, using a number of bespoke file formats, making it difficult to maintain and far from optimal. Additionally, the introduction of the new Cray XC40 High Performance Computer (HPC) at the Met Office promises to deliver a step change in computing performance, leading to the risk that downstream product generation systems will be unable to cope with the increased data volumes, and the benefits of the planned MetUM enhancements will not be fully realised.

To address these concerns, a project to standardise the science and “packaging” of gridded post-processing was initiated. This aims to decouple (upstream) model changes from (downstream) customer-specific products and introduce the concept of a processing structure, enabling the governance required to comprehensively describe and manage the provision of gridded forecast data. The software comprises of a novel use of Python with MOSciTools modules on a High Performance Computing (HPC) Platform. The project should significantly reduce the complexity of the production chain.

This paper will report on the development of the initial version of the software framework which underpins it and the challenges to be addressed to deliver an initial operational capability on the HPC.