Top-down vs bottom-up pose estimation

The two paradigms#

Top-down factorises the problem as detection followed by single-person pose estimation:

$$ p(\text{poses} \mid I) = \prod_{i=1}^{N} p(\text{pose}_i \mid \text{crop}(I, b_i)) \cdot p({b_i} \mid I) $$

Inference cost is $C_{\text{det}} + N \cdot C_{\text{pose}}$. Accuracy benefits from scale normalisation: each crop is resized to a fixed input, so the model always sees a person at roughly constant scale.

Bottom-up predicts all keypoints jointly, then partitions them. Cost is $C_{\text{heat}} + C_{\text{group}}$, with $C_{\text{group}}$ near-constant in $N$.

Grouping methods#

Part Affinity Fields. Cao et al. (CVPR 2017; TPAMI 2021), OpenPose (arxiv.org/abs/1812.08008), predict a 2D vector field $\mathbf{L}c$ per limb type $c$. For a candidate connection between joints at $\mathbf{d}{j_1}$ and $\mathbf{d}_{j_2}$:

$$ E = \int_{u=0}^{1} \mathbf{L}c\big(\mathbf{p}(u)\big) \cdot \frac{\mathbf{d}{j_2} - \mathbf{d}_{j_1}}{\lVert \mathbf{d}_{j_2} - \mathbf{d}_{j_1} \rVert_2}\, du, \qquad \mathbf{p}(u) = (1-u)\mathbf{d}_{j_1} + u\,\mathbf{d}_{j_2} $$

The integral is approximated by uniform sampling. The full assignment over all joints is NP-hard, so OpenPose relaxes it to independent bipartite matchings per limb type, solved by the Hungarian algorithm, then merges limbs sharing a joint. The relaxation is the source of its characteristic crowd failures.

Associative embedding. Newell et al. (NeurIPS 2017) (arxiv.org/abs/1611.05424) predict a scalar tag per keypoint alongside the heatmap, trained so that tags of one person are close and tags of different people are far:

$$ L_{\text{pull}} = \frac{1}{NK}\sum_{n}\sum_{k} (h_k(x_{nk}) - \bar{h}n)^2, \qquad L{\text{push}} = \frac{1}{N^2}\sum_{n}\sum_{n'} \exp\left(-\frac{1}{2\sigma^2}(\bar{h}n - \bar{h}{n'})^2\right) $$

$h_k$ is the tag map for joint $k$, $x_{nk}$ person $n$'s joint $k$ location, $\bar{h}_n$ the mean tag for person $n$. Grouping is then a greedy 1D clustering over tag values. This underpins HigherHRNet (Cheng et al., CVPR 2020), which adds a higher-resolution deconvolution branch specifically to handle small people, the bottom-up weakness.

PersonLab / offset regression. Papandreou et al. (ECCV 2018) predict, at every keypoint, offsets to every other keypoint of the same person, then greedily decode from the most confident seed. Grouping becomes regression rather than combinatorial search.

The scale problem#

Bottom-up processes the whole image at one scale, so a person occupying 5% of the frame and one occupying 80% share a feature representation. Top-down normalises this away by construction.

Bottom-up responses:

• Multi-scale testing — accurate, and it destroys the constant-cost advantage.
• Higher-resolution output — HigherHRNet's stride-2 branch.
• Scale-aware supervision — per-person adaptive $\sigma$ in the target heatmaps.

The gap on COCO has narrowed and has not closed. Top-down still leads at the high-accuracy end.

Current numbers, with the usual caveats#

RTMPose (Jiang et al., 2023, arxiv.org/abs/2303.07399) is the practically important entry: a top-down method fast enough on CPU that the top-down cost argument largely dissolves for small crowds. Reported throughput is hardware-specific; treat the ordering as reliable.

Single-stage methods#

A third family predicts complete poses directly, avoiding both the detector and the grouping step.

DEKR (Geng et al., CVPR 2021) uses disentangled keypoint regression with adaptive convolutions. YOLO-Pose (Maji et al., 2022) attaches keypoint regression heads to a YOLO detector and optimises OKS directly, so one forward pass yields boxes and poses together. PETR and ED-Pose apply DETR-style set prediction, removing NMS as well.

Single-stage methods sit between the two paradigms on both cost and accuracy, and they are attractive when a person detector is needed anyway.

Choosing, in practice#

The decision is dominated by the expected number of people per frame and the accuracy floor:

• $N = 1$, accuracy critical → top-down, or a dedicated single-person model.
• $N$ small and bounded, CPU budget → RTMPose-class top-down.
• $N$ large or unbounded, fixed latency → bottom-up, and accept crowd errors.
• Detector already in the pipeline → single-stage, which reuses it.

Measure $C_{\text{det}}$, $C_{\text{pose}}$ and $C_{\text{heat}}$ on your hardware before committing. The crossover moves by a factor of several between a laptop CPU and a datacentre GPU.

Papers#

• Cao et al., Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, CVPR 2017 / TPAMI 2021 — arxiv.org/abs/1812.08008
• Newell et al., Associative Embedding, NeurIPS 2017 — arxiv.org/abs/1611.05424
• Papandreou et al., PersonLab, ECCV 2018 — arxiv.org/abs/1803.08225
• Sun et al., HRNet, CVPR 2019 — arxiv.org/abs/1902.09212
• Cheng et al., HigherHRNet, CVPR 2020 — arxiv.org/abs/1908.10357
• Geng et al., Bottom-Up Human Pose Estimation via Disentangled Keypoint Regression, CVPR 2021 — arxiv.org/abs/2104.02300
• Jiang et al., RTMPose, 2023 — arxiv.org/abs/2303.07399

What to learn next#

• Object detection — the first half of every top-down pipeline.
• YOLO — the detector most pose pipelines actually use, and the base for YOLO-Pose.
• Latency and throughput — measuring the constants the crossover depends on.