RoI pooling and RoI align

RoI pooling#

Given a region $(x_1, y_1, x_2, y_2)$ in image coordinates and a feature map at stride $s$, RoI pooling quantises twice:

$$ \tilde{x} = \left\lfloor \frac{x}{s} \right\rfloor, \qquad \text{bin}_{ij} = \left[ \tilde{x}_1 + \left\lfloor \frac{i \cdot \tilde{w}}{K} \right\rfloor,\; \tilde{x}_1 + \left\lceil \frac{(i+1) \cdot \tilde{w}}{K} \right\rceil \right) $$

Where $K$ is the output grid size and $\tilde w = \tilde x_2 - \tilde x_1$. Each bin is reduced by max. The first quantisation snaps the RoI to the feature grid; the second snaps the bin boundaries.

The first rounding misaligns each edge by up to $s/2$ pixels. The second adds up to $s$ per bin boundary. At $s = 16$ that is a substantial fraction of a small object.

Crucially, $\partial \text{output} / \partial x_1 = 0$ almost everywhere, because the floor function has zero derivative. RoI pooling gives no gradient with respect to box coordinates.

RoI align#

He et al. (2017) remove both quantisations. Bin boundaries stay in continuous coordinates. $N$ sample points sit regularly inside each bin, and each is read by bilinear interpolation:

$$ f(x, y) = \sum_{i,j} f(x_i, y_j) \max(0, 1 - |x - x_i|) \max(0, 1 - |y - y_j|) $$

Bin values are the average over sample points. Bilinear interpolation is piecewise differentiable in $x$ and $y$, so gradients flow to box coordinates as well as to features. That property is what makes fully differentiable two-stage detectors and iterative box refinement possible.

The reported effect in the Mask R-CNN paper is large for masks and modest for boxes. It grows as the stride grows. The gap is widest on the coarse pyramid levels, which is where the quantisation error in pixels is largest.

The aligned half-pixel correction#

Continuous coordinate $x$ maps to feature coordinate $x/s$. If pixel centres are taken to lie at half-integers, the correct mapping is $x/s - 0.5$. Omitting the shift biases every sample by half a feature cell.

Detectron2 exposed this as aligned=True; torchvision follows. The reported effect is a small but consistent AP gain, larger for masks. It costs nothing at runtime.

Alternatives#

• RoI warp (Dai et al., 2016) uses a differentiable affine warp. It is close in spirit to align, formulated through a spatial transformer.
• PrRoI pooling (Jiang et al., 2018) integrates the bilinear surface over each bin in closed form. Averaging discrete samples is no longer needed. That removes the sampling_ratio hyperparameter and gives continuous gradients.
• Deformable RoI pooling (Dai et al., 2017) learns per-bin offsets, letting bins move off the rectangle to follow the object.
• Deformable attention (Zhu et al., 2021) generalises this: a query predicts a small set of sampling offsets and attention weights over multi-scale features. This is the mechanism that replaced RoI pooling entirely in the DETR line; see DETR and set prediction.

Cost#

Per RoI: $O(C \cdot K^2 \cdot N)$ where $C$ is channels, $K$ the output grid and $N$ the samples per bin. With $C=256$, $K=7$, $N=4$, that is around 50,000 reads per RoI. At 512 RoIs per image in training, RoI align is a real fraction of the forward pass. It is memory-bandwidth-bound rather than compute-bound.

This cost is the structural reason one-stage detectors are faster: they never crop. The dense head is applied convolutionally to the whole map once.

Papers#

• Girshick, Fast R-CNN, ICCV 2015 — arxiv.org/abs/1504.08083
• He et al., Mask R-CNN, ICCV 2017 — arxiv.org/abs/1703.06870
• Jiang et al., Acquisition of Localization Confidence for Accurate Object Detection (PrRoI pooling), ECCV 2018 — arxiv.org/abs/1807.11590
• Dai et al., Deformable Convolutional Networks, ICCV 2017 — arxiv.org/abs/1703.06211

What to learn next#

• R-CNN, Fast R-CNN and Faster R-CNN — the detectors this operation was built for.
• Feature pyramid networks — which level each region gets cropped from.
• Image segmentation — where the half-pixel accuracy actually shows.