Challenges with Machine Learning Interpretability as shown by a Climate Study

Abstract:

Progress has been made in interpreting machine learning (ML) models, helping to push back on the notion that they are “black boxes.” Some interpretability techniques are relatively straightforward to implement and understand, such as permutation feature importance and saliency maps. Permutation feature importance involves randomizing input features to quantify their respective impact on error, and saliency maps are computed using the gradient of the output with respect to the input features, where higher magnitude gradients represent areas of greater importance. While there is insight gained from these interpretability techniques, results can be misleading due to sensitivities of the analyses to skill metrics and data partitioning.

Permutation feature importance and saliency maps were used to interpret the reasons for skillful results generated by a convolutional neural network that was trained to classify strongly-rotating convective storms extracted from a high-resolution climate simulation over the United States. Here we show that permutation feature importance exhibits sensitivity to the chosen error metric, along with the population balance between majority and minority classes during training, resulting in different physical variables being denoted as important. On spatial interpretation, saliency maps can help visualize the spatial areas of features that activate for specific model predictions, but it can be difficult to extract a meaningful signal from a large volume of maps that describe an ML model’s decision output, which can lead to cherry-picking of results that describe what humans believe to be meaningful and physical. Therefore, while these interpretation methods can help humans understand the reasons for ML model outputs, they are susceptible to confirmation bias, where humans may report back interpretability results that agree with prior knowledge. This challenge extends beyond potentially misleading results produced by individual ML studies, as confirmation bias during interpretability can hinder ML-driven knowledge discovery in Earth system science. Potential ways to reduce confirmation bias will be discussed, including hypothesis pre-registration, training with reduced input features, and exploring saliency map variability of similar samples in the latent space.