Grad-CAM and saliency maps

Class activation mapping#

Zhou et al. (2016) introduced CAM for architectures ending in global average pooling followed by one linear layer. With final feature maps $A^k$ and classifier weight $w_k^c$,

$$ M_c(i,j) = \sum_k w_k^c \, A^k(i,j) $$

The restriction is severe: the architecture must have that exact tail, and models without it had to be modified and retrained.

Selvaraju et al. (2017/2020) removed it. Grad-CAM replaces the classifier weight with a gradient average:

$$ \alpha_k^c = \frac{1}{Z}\sum_{i}\sum_{j} \frac{\partial y^c}{\partial A^k(i,j)}, \qquad L_{\text{Grad-CAM}}^c = \mathrm{ReLU}!\left(\sum_k \alpha_k^c A^k\right) $$

$y^c$ is the pre-softmax score for class $c$, $Z$ the number of spatial positions. For a GAP-plus-linear network $\alpha_k^c$ reduces exactly to $w_k^c / Z$, so Grad-CAM is a strict generalisation of CAM.

The ReLU keeps only features with a positive influence on $y^c$. Removing it produces a signed map that is often more informative and is not what the paper defines.

The global spatial average in $\alpha_k^c$ is the source of the failure demonstrated in the developer block: all spatial structure in the gradient is discarded, so class specificity must come entirely from the channels.

Variants, and what each fixes#

HiResCAM deserves particular attention. Draelos and Carin prove that Grad-CAM's spatial averaging can produce maps highlighting regions that provably did not contribute, and that the element-wise product $\sum_k \frac{\partial y^c}{\partial A^k} \odot A^k$ is faithful for architectures whose head is a single fully-connected layer after the explained layer. The developer example above is a minimal instance of the failure their proof describes.

Sanity checks#

Adebayo et al. (2018), Sanity Checks for Saliency Maps, propose two randomisation tests:

• Model parameter randomisation. Reinitialise the network's weights layer by layer, from the top down, and measure map similarity to the original. A method whose maps survive randomisation is not a function of what the model learned.
• Data randomisation. Retrain on permuted labels. A method whose maps are unchanged is not a function of the label relationship.

Guided Backpropagation and Guided Grad-CAM fail both, behaving essentially as edge detectors. Plain Grad-CAM and input gradients pass the parameter test. The result is why "it looks like the object" is not evidence for a saliency method.

Kindermans et al. (2019), The (Un)reliability of Saliency Methods, add an input-invariance test: adding a constant to every input, absorbed by an adjusted bias so the model's function is unchanged, alters the maps produced by several popular methods.

Quantitative evaluation#

Since ground-truth explanations do not exist, evaluation is comparative and causal.

Deletion and insertion (Petsiuk et al., 2018, RISE): progressively remove the highest-ranked pixels and plot class probability against fraction removed; the area under the deletion curve should be low, and under the insertion curve high. Both are confounded by distribution shift — a blacked-out region is itself out of distribution — which Hooker et al. (2019), ROAR, address by retraining on the modified data at each threshold. ROAR found several popular methods no better than a random ranking.

Pointing game (Zhang et al., 2018): the fraction of images where the map's maximum falls inside a ground-truth bounding box. Cheap, and only weakly related to faithfulness.

Attribution beyond CAMs#

Integrated Gradients (Sundararajan et al., 2017) integrates the gradient along a straight path from a baseline $x'$ to the input:

$$ \mathrm{IG}_i(x) = (x_i - x'i) \int{0}^{1} \frac{\partial F(x' + \alpha(x - x'))}{\partial x_i} \, d\alpha $$

It satisfies completeness (attributions sum to $F(x) - F(x')$) and implementation invariance, and it requires choosing a baseline. A black image is the usual choice and is a poor one for any task where black is meaningful.

SHAP and LIME provide model-agnostic alternatives with different guarantees; see SHAP and LIME. For transformers, raw attention weights are not attributions — Jain and Wallace (2019) and Wiegreffe and Pinter (2019) conducted the debate; attention rollout and gradient-weighted attention (Chefer et al., 2021) are the usable versions.

The honest position#

Saliency methods are hypothesis generators. They have repeatedly located genuine dataset artefacts — chest-radiograph models keying on hospital-specific tokens, dermatology models keying on rulers placed beside malignant lesions — and that is a real contribution. They do not establish which features a model used, they disagree with each other on the same input, and several fail basic sanity checks. Any claim resting on a heat map should be confirmed by intervention: occlude, retrain without the region, or test on data where the suspected artefact is absent.

Papers#

• Zhou et al., Learning Deep Features for Discriminative Localization, CVPR 2016 — arxiv.org/abs/1512.04150
• Selvaraju et al., Grad-CAM, ICCV 2017 / IJCV 2020 — arxiv.org/abs/1610.02391
• Sundararajan et al., Axiomatic Attribution for Deep Networks, ICML 2017 — arxiv.org/abs/1703.01365
• Adebayo et al., Sanity Checks for Saliency Maps, NeurIPS 2018 — arxiv.org/abs/1810.03292
• Petsiuk et al., RISE: Randomized Input Sampling for Explanation, BMVC 2018 — arxiv.org/abs/1806.07421
• Hooker et al., A Benchmark for Interpretability Methods in Deep Neural Networks (ROAR), NeurIPS 2019 — arxiv.org/abs/1806.10758
• Draelos and Carin, Use HiResCAM instead of Grad-CAM, 2020 — arxiv.org/abs/2011.08891
• Chefer et al., Transformer Interpretability Beyond Attention Visualization, CVPR 2021 — arxiv.org/abs/2012.09838

What to learn next#

• Explainability — the broader question these maps are one answer to.
• SHAP and LIME — model-agnostic attribution with different guarantees.
• Convolutional neural networks — the feature maps Grad-CAM is reading.