Combined Machine Learning-Ready Benchmark Datasets of Solar Energetic Phenomena

A recent roadmap review^[1] on the prediction of solar energetic phenomena (flares, coronal mass ejections [CMEs], and solar energetic particle [SEP] events) proposed a new conceptual framework able to significantly help advance our space weather forecast capability. The framework treats a given forecast method, be it a conventional or machine/deep learning one, as a “theory” that uses a set of intuitive concepts and ideas and, importantly, creates testable predictions.

To test a method’s predictions, one needs (i) data for training, validation and testing; (ii) a forecast model in which the idea is implemented; and (iii) a verification strategy, for all involved items: data, model, and model performance. Keeping the data and verification strategy fixed, the only variable element is the model, so different models can be trained, validated and tested under the exact same conditions. This approach allows accurate comparison between different models, leading tangibly into advances in our overall forecast capability or, more likely, in the realization that we need better forecast models. This conceptually straightforward task has only been achieved in less than a handful of cases for the forecasting of solar events, with enlightening results.

We discuss here the first – arguably most important – step in realizing such a framework, namely, the creation of datasets to which each candidate methods can be applied. The resulting datasets require demonstrated statistical significance, detailed curation, the removal of potential outliers or artifacts, homogeneity via elimination of data gaps, etc. In this case, one creates benchmark datasets for a given forecast problem. These datasets can be augmented either “horizontally” or “vertically” to achieve better statistics for the problem at hand or expand the range of problems they can be applied to, respectively.

We present openly accessible benchmark datasets for all forecast problems of the explosive and eruptive solar weather, namely, flares, CMEs and SEP events. The link between them is their focus on the solar sources (namely, solar active regions) before they erupt. These datasets correspond to solar flares^[2,3], magnetic polarity inversion lines^[4], integration between flares and CMEs^[5], and SEP events^[6]. Given that these datasets have received similar levels of curation, they can be combined in a vertical fashion to take on more problems. We provide some examples on this.

The capabilities of these and similar datasets are substantial. Moreover, the datasets are compatible by design with machine (and deep) learning methodologies. Different conventional and AI methodologies can be validated against each other to precisely realize the gain brought by these innovative new techniques.

We argue that data benchmarking is a necessary step in advancing our forecasting capabilities for any group of space-weather-related problems. Self-consistency in this case can also boost physical understanding of the forecast results, as well as physical interpretability in case unsupervised methodologies are used. These are necessary actions for achieving a functional research-to-operations to research (R2O2R) loop, that is effectively an optimization loop.

Our presentation aims to inform and engage the community on these topics, showing also that interdisciplinarity is more of a necessity than a luxury in achieving a sustainable and optimizable forecast of the solar end of space weather.

[1] Georgoulis, M. K., Yardley, S. L., Guerra, J. A. et al.: Prediction of Solar Energetic Events Impacting Space Weather Conditions, Advances Space Res., 2024, in press (doi: https://doi.org/10.1016/j.asr.2024.02.030)

[2] Angryk, R. A., Martens, P. C., Aydin, B., et al.: Multivariate Time Series Dataset for Space Weather Data Analytics, Scientific Data, 7:227, 2020 (doi: https://doi.org/10.1038/s41597-020-0548-x)

[3] Pandey, C. & Aydin, B., Bitmap Filtered Line of Sight HMI Active Region Patches with Augmentations, Harvard Dataverse, 2024 (doi: https://doi.org/10.7910/DVN/WLTBQQ)

[4] Ji, A., Cai, X., Khasayeva, N., Georgoulis, M. K., Martens, P. C., Angryk, R. A. & Aydin, B.: A Systematic Magnetic Polarity Inversion Line Data Set from SDO/HMI Observations, Astrophys. J. Suppl. Series, 265:28, 2023 (doi: https://doi.org/10.3847/1538-4365/acb43a)

[5] Ji, A. & Aydin, B.: Flare to CME Association Integration, Harvard Dataverse, 2022 (doi: https://doi.org/10.7910/DVN/WSEY4T)

[6] Rotti, S., Aydin, B., Georgoulis, M. K. & Martens, P. C.: Integrated Geostationary Solar Energetic Events Catalog: GSEP, Astrophys. J. Suppl. Series, 262:29, 2022 (doi: https://doi.org/10.3847/1538-4365/ac87ac)