Exploring Downscaling and Upscaling Using the Transformer Decoder:A Case Study with a High-Dimensional Generalized Lorenz Model

While the groundbreaking findings within the Lorenz 1963 model suggested finite predictability, new insights from the development of the generalized Lorenz model (Shen 2019; Shen et al. 2021; 2022c) and the revisit of the Lorenz 1969 model (Shen et al. 2022a) indicate distinct predictability for chaotic and regular solutions. Additionally, our reviews (Shen et al. 2023; 2024) have reiterated that the well-accepted two-week predictability limit was extrapolated from a doubling time of 5 days within a general circulation model. It was not established by Lorenz's models (Shen 2023). As a result, we suggested referring to the two-week predictability limit as the Predictability Limit Hypothesis, acknowledging the collective contribution from Lorenz, Mintz, and Arakawa under the leadership of Charney, and drawing a parallel to Moore's law, which is not a physics-based law. The new insights provide a stronger foundation for exploring predictability horizons.

Historically, although the negative impact of the butterfly effect has been a focus in predictability studies (Shen et al. 2022b), the positive role of multiscale processes in determining predictability within real-world global models has been a focus since the late 2000s (e.g., Shen et al. 2010a, b; Wu and Shen 2016). Interestingly, the role of multiscale processes enabled by the high-resolution global models is analogous to that in the long-range dependencies enabled by the self-attention mechanisms of recent large language models such as ChatGPT. This type of analogy is also supported by the fact that many AI-powered models, some of which produce short-term simulations better than the state-of-the-art PDE-based models, were constructed by the Vision Transformer, i.e., a variant of transformer technology.

By leveraging the power of machine learning (ML) techniques, particularly Transformer technology, this study examines the validity of a conceptual model involving downscaling and upscaling to determine important factors for predictability (Shen et. al. 2010a, b). Providing a proof of concept, chaotic data from our recently-developed generalized Lorenz model are used to train and validate ML-based empirical models. These empirical models are developed to be flexible, allowing the choice of different input variables (e.g., wind or temperature fields) and output variables (e.g., precipitation) to establish statistical relationships between them.

Specifically, we employed the high-dimensional generalized Lorenz model (GLM) to generate chaotic data across multiple scales. This data was then used to train three types of machine learning models: a linear regression model, a FeedForward Neural Network (FFNN) model, and a transformer-based model. The linear regression model served as a foundational approach by using large-scale variables to predict small-scale variables. In contrast, the FFNN and transformer-based models incorporated additional complexity through multiple hidden layers and self-attention mechanisms, respectively, to enhance prediction accuracy. All three models demonstrated robust performance, with correlation coefficients between the predicted and actual small-scale variables exceeding 0.9. Notably, the transformer-based model, which yielded better results than the others, exhibited strong performance in both control and parallel runs, where sensitive dependence on initial conditions (SDIC) occurs during the validation period. This study highlights that large-scale variables, which retain a memory of their connection to smaller scales, can be effectively utilized by trained empirical models to estimate small-scale variables. Future research could investigate the interactions between large-scale variables (e.g., a Madden-Julian Oscillation or an atmospheric river) and small-scale processes (e.g., precipitation).

References:

Shen, B.-W., 2019: Aggregated Negative Feedback in a Generalized Lorenz Model. International Journal of Bifurcation and Chaos, Vol. 29, No. 3 (2019) 1950037 (20 pages). https://doi.org/10.1142/S0218127419500378

Shen, B.-W., 2023: A Review of Lorenz's Models from 1960 to 2008. International Journal of Bifurcation and Chaos. Vol. 33, No. 10, 2330024. https://doi.org/10.1142/S0218127423300240.

Shen, B.-W., 2024: Exploring Downscaling in High-Dimensional Lorenz Models Using the Transformer Decoder. Available from ResearchGate: https://doi.org/10.13140/RG.2.2.18465.21603 (submitted)

Shen, B.-W., W.-K. Tao, and M.-L. Wu, 2010b: African Easterly Waves in 30-day High-resolution Global Simulations: A Case Study during the 2006 NAMMA Period. Geophys. Res. Lett., 37, L18803, doi:10.1029/2010GL044355.

Shen, B.-W., W.-K. Tao, W. K. Lau, R. Atlas, 2010a: Predicting Tropical Cyclogenesis with a Global Mesoscale Model: Hierarchical Multiscale Interactions During the Formation of Tropical Cyclone Nargis (2008) . J. Geophys. Res.,115, D14102, doi:10.1029/2009JD013140.

Shen, B.-W., R. A. Pielke Sr., X. Zeng, J.-J. Baik, S. Faghih-Naini, J. Cui, and R. Atlas, 2021: Is Weather Chaotic? Coexistence of Chaos and Order within a Generalized Lorenz Model. Bulletin of the American Meteorological Society, 102(1), E148-E158. https://doi.org/10.1175/BAMS-D-19-0165.1

Shen, B.-W., R. A. Pielke Sr., X. Zeng, J. Cui, S. Faghih-Naini, W. Paxson, A. Kesarkar, X. Zeng, R. Atlas, 2022c: The Dual Nature of Chaos and Order in the Atmosphere. Atmosphere 13, no. 11: 1892. https://doi.org/10.3390/atmos13111892.

Shen, B.-W., R. A. Pielke Sr., X. Zeng, J. Cui, S. Faghih-Naini, W. Paxson, R. Atlas, 2022b: Three Kinds of Butterfly Effects Within Lorenz Models. Encyclopedia 2, no. 3: 1250-1259. https://doi.org/10.3390/encyclopedia2030084

Shen, B.-W., R. A. Pielke Sr., X. Zeng, 2022a: One Saddle Point and Two Types of Sensitivities Within the Lorenz 1963 and 1969 Models. Atmosphere 13, no. 5: 753. https://doi.org/10.3390/atmos13050753

Shen, B.-W., R. A. Pielke Sr., X. Zeng, and X. Zeng, 2023: Lorenz’s View on the Predictability Limit. Encyclopedia 2023, 3(3), 887-899; https://doi.org/10.3390/encyclopedia3030063

Shen, B.-W., R. A. Pielke Sr., X. Zeng, and X. Zeng, 2024: Exploring the Origin of the Two-Week Predictability Limit: A Revisit of Lorenz’s Predictability Studies in the 1960s. Atmosphere, 15(7), 837; https://doi.org/10.3390/atmos15070837 (A review of AI-powered models is provided)

Wu, Y.-L.,and B.-W. Shen, 2016: An Evaluation of the Parallel Ensemble Empirical Mode Decomposition Method in Revealing the Role of Downscaling Processes Associated with African Easterly Waves in Tropical Cyclone Genesis. J. Atmos. Oceanic Technol. 33, 1611-1628, DOI: 10.1175/JTECH-D-15-0257.1.