Face alignment and landmarks

The problem, stated#

Given detected landmarks ${p_i}_{i=1}^{n} \subset \mathbb{R}^2$ and a canonical template ${q_i}_{i=1}^{n}$, find the similarity transform minimising

$$ E(s, R, t) = \frac{1}{n}\sum_{i=1}^{n} \lVert q_i - (sRp_i + t) \rVert^2 $$

$s > 0$ is a scalar, $R \in SO(2)$ a rotation, $t \in \mathbb{R}^2$ a translation. The restriction to $SO(2)$ rather than $O(2)$ excludes reflections, and to a single scalar $s$ rather than a diagonal matrix excludes anisotropic scaling.

Umeyama (1991) gives the closed-form solution. With $\mu_p, \mu_q$ the centroids and $\Sigma = \frac{1}{n}\sum (q_i - \mu_q)(p_i - \mu_p)^\top$ the cross-covariance, take the SVD $\Sigma = UDV^\top$ and set

$$ R = USV^\top, \qquad s = \frac{\operatorname{tr}(DS)}{\sigma_p^2}, \qquad t = \mu_q - sR\mu_p $$

where $\sigma_p^2 = \frac{1}{n}\sum \lVert p_i - \mu_p \rVert^2$ and

$$ S = \begin{cases} I & \text{if } \det(U)\det(V) \geq 0 \ \operatorname{diag}(1, -1) & \text{otherwise} \end{cases} $$

$S$ is the reflection guard. Its omission is a recurring bug in reimplementations, and it fails silently on well-conditioned data.

Reference: Umeyama, Least-Squares Estimation of Transformation Parameters Between Two Point Patterns, IEEE TPAMI 13(4), 1991.

The canonical template#

The five-point template used above is the arcface_src array shipped with InsightFace, for a $112 \times 112$ crop. It is not derived from first principles; it is the mean landmark configuration of a large frontal training set, and it is a convention that the weights depend on.

This makes the template part of the model interface. Two systems using different templates produce embeddings that are not comparable, even with identical architectures and losses.

Alignment sensitivity#

The relationship between landmark error and recognition accuracy is steep and well documented. Deng et al. (2020) report that landmark supervision inside RetinaFace improves downstream verification accuracy on IJB-C, attributing the gain to alignment quality rather than detection recall.

The mechanism is a train/test distribution shift. A recognition model trained on crops aligned with a particular detector inherits that detector's landmark bias. Swapping detectors at deployment shifts the crop statistics, and accuracy falls without any component being individually broken.

ARoFace (Ebrahimi et al., ECCV 2024, arxiv.org/abs/2407.14972) attacks this by treating alignment perturbation as an augmentation during recognition training, improving robustness on low-quality benchmarks where landmark estimates are unreliable.

Dense landmarks#

Five points suffice for a similarity transform. Denser sets serve other purposes:

• 68 points (iBUG 300-W, Sagonas et al., 2013) — the long-standing academic convention, giving jaw, brow, eye, nose and lip contours. Estimated by cascaded regression (Kazemi and Sullivan, CVPR 2014, the ensemble-of-regression-trees method behind dlib) or by heatmap CNNs.
• 468 or 478 points (MediaPipe Face Mesh) — a topologically fixed mesh, used for effects, expression and gaze rather than alignment.
• 3D morphable model fits (3DMM; Blanz and Vetter, 1999) — parametric shape and expression coefficients, enabling frontalisation of non-frontal faces.

Note the trade-off. Dense landmarks make expression and gaze available, and they are noisier per point than the sparse five. For a similarity transform, five well-estimated points beat sixty-eight poorly estimated ones.

Heatmap versus direct regression#

Landmark models split the same way pose models do. Direct coordinate regression is fast and has no quantisation error but generalises less well. Heatmap regression is more accurate and pays a resolution cost, because a coordinate read back from a low-resolution heatmap is snapped to that grid. The sub-pixel decoding problem this creates is worked through later in this section.

Alignment-free alternatives#

Two lines of work question whether alignment is needed at all.

Spatial transformer networks (Jaderberg et al., NeurIPS 2015) learn the warp end to end inside the recognition network. Elegant, and in practice unstable to train for faces, because the warp and the identity objective interact badly early in training.

Larger, augmentation-heavy training sets reduce the alignment gain considerably. Modern high-capacity models trained with aggressive geometric augmentation lose less from imperfect alignment than 2018-era models did. The gain has narrowed; it has not vanished, and it remains largest exactly where accuracy matters most — low-resolution, non-frontal, poorly lit input.

Papers#

• Umeyama, Least-Squares Estimation of Transformation Parameters, TPAMI 1991
• Blanz and Vetter, A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH 1999
• Sagonas et al., 300 Faces in-the-Wild Challenge, ICCV Workshops 2013
• Kazemi and Sullivan, One Millisecond Face Alignment with an Ensemble of Regression Trees, CVPR 2014
• Jaderberg et al., Spatial Transformer Networks, NeurIPS 2015 — arxiv.org/abs/1506.02025
• Ebrahimi et al., ARoFace: Alignment Robustness to Improve Low-Quality Face Recognition, ECCV 2024 — arxiv.org/abs/2407.14972

What to learn next#

• How images are stored — pixels, channels and the layout every warp operates on.
• OpenCV — warpAffine, interpolation modes and the rest of the geometry toolkit.
• Linear algebra — the SVD that makes the closed-form solution possible.