Hierarchical Superpixel Explanations Improve XAI Attribution Methods for Geospatial Models

Complex AI models can learn non-linear relationships to achieve high performance, but their complexity makes it very difficult to determine what they have learned. Exposing the model’s learned strategies may provide opportunities for increasing trust in the model, identifying potential directions for improvements, and even possibly gain new insights into the underlying real-world phenomena. This has motivated the widespread adoption of explainable artificial intelligence (XAI) techniques that probe models to explain what they have learned. However, correlated input features may make it challenging to use XAI methods to learn about either the model or the underlying data relationships. Atmospheric models commonly use gridded geospatial data as inputs such as satellite imagery and numerical weather predictions (NWPs). Geospatial data is commonly characterized by strong correlations, especially autocorrelation between nearby grid cells. In this research, we investigate the influence that correlated input features in geospatial data have on XAI attribution methods. Our study is the first to quantify the effects of strong correlation on XAI results, investigate the reasons that underlie these effects and offer promising strategies to address them. We investigate how correlated input features influence attribution methods in two phases. First, we develop a suite of synthetic benchmarks to analyze how strongly correlated inputs affect attributions for multilayer perceptron (MLP) networks. Second, we explore how the scale of the learned features affects attributions for Convolution Neural Networks (CNNs).

XAI methods are applied to investigate the trained networks, and to gain insights about the real-world physical processes. Correlated features can negatively affect XAI's faithfulness to the network, i.e., make it difficult for XAI to accurately identify the relative importance of the features that the network used to predict. Correlated features can also increase the chances for a network to learn many different feature combinations to achieve approximately the same performance. In the first case, inaccurate attribution maps make it difficult to use XAI for learning about either the model or the data. In the second case, an explanation that accurately captures the network could be difficult to interpret: features that are meaningful for the real-world phenomena may be assigned minimal attribution because the network has learned to exploit another feature that it is correlated with.

In the first phase, we use a benchmark framework consisting of synthetic datasets with increasingly strong correlation, for which the ground truth attribution is known. For each dataset, we train multiple MLP networks to compare XAI-derived attributions to the ground truth and among the set of trained networks. We demonstrate that feature correlation can strongly increase the variance of the derived attributions, which is highly problematic for using XAI to learn about relationships between the data and the physical processes. We then investigate the cause of the high variance: is it because different trained networks learn highly different functions to solve the synthetic prediction task or because the XAI methods themselves become less faithful in the presence of strong correlation? Finally, we show that applying XAI on superpixels, instead of single grid cells, substantially increases attribution agreement. In this case, we are able to identify that strong autocorrelation provides many opportunities for the networks to learn strategies that are different among a small neighborhood of grid cells.

In the second phase, we use satellite and NWP datasets to explore how the scale of the learned features affects CNN attributions. CNNs extract spatial patterns from the gridded input. To detect clouds, for example, the CNN needs to consider shape and texture. Since single pixels are not informative in isolation, applying attribution methods to explain individual grid cells may not reveal the spatial patterns that drive the model. Instead, XAI methods can be applied to groups of grid cells. We demonstrate that the partitioning of the input into grouped features has a considerable impact on the attributions. Very often, our results show that the grid cell-level attributions are noisy and fail to reveal the spatial features that the model relies on. As the input and convolutional kernel sizes increase, so does the tendency for the model to learn larger-scale features that require increasingly larger superpixel sizes to reveal. After exposing this issue, we propose strategies using hierarchical clustering of the input data to generate data-driven superpixels to achieve more useful XAI results.

Our results highlight that applying off-the-shelf XAI methods can produce very misleading explanations. Relying on these explanations could be worse than treating the model as a black box. However, we propose strategies to detect evidence that these issues are occurring as well as potential solutions based on training multiple models and applying attribution methods to a hierarchy of grid cell clusters to identify the spatial patterns that the model uses.