TWXT: Integrating Text and Numerical Weather Prediction Data for Advanced Weather Forecasting

This study introduces TWXT, a novel framework that leverages machine learning (ML) to integrate text-based weather reports with numerical weather prediction data, expanding the capabilities of ML in both weather forecasting and text generation. Our multi-faceted approach began by training transformer models on Storm Prediction Center (SPC) convective outlooks, achieving near-perfect accuracy in predicting the author of the outlook from the text.

Central to our current research is the creation of a unique paired dataset that combines numerical weather data with textual descriptions. We incorporate data from the Community Collaborative Rain, Hail, and Snow Network (CoCoRaHS), a citizen science initiative providing dense precipitation observations across the United States. Approximately 10% of the collected data included textual descriptions, which we cleaned using standard text processing techniques, including Levenshtein distance matching against predefined weather terms. We then paired this processed text data with corresponding Rapid Update Cycle (RAP) analysis soundings over the 24-hours prior to the report, creating a rich dataset of (RAP sounding, text description) pairs.

Leveraging this paired dataset, we explore advanced hybrid methods to process 24 hours of RAP soundings and describe associated weather conditions. Specifically, we adapt the CLIP (Contrastive Language-Image Pre-training) methodology to learn joint embeddings of 24-hour RAP soundings paired with textual descriptions of daily weather. This approach enables us to explore zero-shot classification capabilities, potentially allowing the model to identify new weather patterns without explicit training. Additionally, we are utilizing Recurrent Neural Networks (RNNs) to encode the 24-hour sounding data at one-hour resolution, with a decoder predicting sequences of weather terms. By incorporating encoder-decoder attention mechanisms, we aim to identify which hourly soundings most significantly impact predicted weather tokens, thereby enhancing interpretability.

Initial results demonstrate the models' proficiency in accurately identifying weather labels and learning meaningful joint embeddings. The attention maps from the RNN approach show promise in highlighting hourly soundings within the 24-hour period that influence specific weather terminology predictions. While these findings are promising, this research is in its early stages, laying the groundwork for more weather prediction and text generation models that integrate numerical and textual data sources. Future work will focus on refining these models, expanding the range of predictable weather phenomena, and advancing fully automated text generation for comprehensive weather forecasts.