Systematic Biases of Microphysics Parameterizations in Precipitation Simulations: A Large Language Model–Based Review of 2699 Publications

Where do the errors in precipitation forecasts and simulations come from? Although imperfect microphysics parameterizations must be near the top of the list, documenting errors in such parameterizations and where they come from is challenging, especially when trying to generalize model errors from single case studies. Also, many studies only use one microphysics parameterization, so intercomparisons between different parameterizations to determine the best performer in a given situation are often not done.

Analyzing a large number of scientific publications that have verified model forecasts and simulations systematically is a potential approach to begin to address these limitations. The hope is that a large database would offset the limitations of unknown sources of the errors, comparisons across parameterizations, and in situ verification, for example. Better understanding of the literature as a whole could reveal systematic biases in the various parameterizations, which parameterizations are better or worse in various parts of the world, and opportunities for further improvements in these parameterizations based on these biases. Yet, to our knowledge, no systematic literature review, meta-analysis, or other study has constructed a large database of literature characterizing the performance of microphysics parameterizations. Opportunities for advances would be ripe were such a database to exist, but manually reading thousands of publications to understand model performance and summarize systematic biases for different microphysics parameterizations would be time consuming and labor intensive. How could such a large database of studies be constructed and be feasibly evaluated?

The solution may lie within the newly emerging field of large language models. We perform a review of 2699 publications associated with precipitation simulations in the Weather and Research Forecasting (WRF) model in search of systematic biases of different microphysics parameterizations. The database was constructed from nearly 4000 publications published prior to 22 June 2023 identified from Web of Science and Scopus searches. These publications were reduced to a subset of 2699 by eliminating coupled WRF simulations, those that did not predict precipitation, and those where the full text was unavailable. The large language model GPT-4 Turbo was used to extract information about model configurations and performance from the entire text of these 2699 publications. We asked GPT-4 Turbo eight questions about these 2699 publications: the configuration of the model and its selection of parameterizations, the simulation domain, and information about the precipitation forecast or simulation (including the variables measured, the degree of overestimation or underestimation, values of root mean squared errors and correlation coefficients).

Our results reveal the landscape of how nine of the most popular microphysics parameterizations have been used around the world (listed here from most to least commonly used): WRF Single-Moment (WSM versions 3, 5, and 6), Thompson, Morrison, Lin, WRF Double-Moment (WDM), Goddard Cumulus Ensemble, and Ferrier. One-moment bulk microphysics parameterizations were more popular than two-moment parameterizations before 2019, and two-moment parameterizations were more popular after 2020. Researchers preferred the Lin, WSM, and WDM parameterizations in south Asia and China, whereas they preferred the Ferrier, WSM6, Thompson, and Morrison parameterizations in the United States. Seven out of nine microphysics parameterizations tended to overestimate precipitation. However, systematic biases of microphysics parameterizations differed in various regions. Simulations using three of the parameterizations tended to underestimate precipitation over almost all locations around the world, whereas the remaining six parameterizations tended to overestimate. Specifically, precipitation over China, southeast Asia, western United States, and central Africa tended to be overestimated, whereas precipitation over the eastern United States tended to be underestimated. For those studies that presented verification statistics of root mean squared errors and correlation coefficients, some parameterizations clearly performed the best and some clearly performed poorly, based on as many as 1000 different verifying simulations within each microphysics parameterization.

Come to the presentation to find out which parameterizations had the best performance statistics.