Image segmentation

The task#

Semantic segmentation is dense classification:

f_theta : R^(H x W x 3)  ->  R^(H x W x K),   prediction  y_hat(i,j) = argmax_k f_theta(x)_{i,j,k}

• H, W — spatial dimensions.
• K — number of classes, including a background or void class.

Instance segmentation additionally partitions pixels into object identities, and panoptic segmentation (Kirillov et al., 2019) unifies both by assigning every pixel a (class, instance_id) pair, with instance_id ignored for "stuff" classes.

The structural tension in every architecture here is resolution against context. Classification backbones downsample aggressively — a stride of 32 is standard — because context requires a large receptive field. Dense prediction needs the output back at full resolution. Every design below is a different answer to that.

Architectural lineage#

FCN (Long, Shelhamer and Darrell, 2015) replaced the classifier's fully connected layers with convolutions and upsampled with learned transposed convolutions, fusing skip connections from earlier, higher-resolution stages. This is the paper that created the field.

U-Net (Ronneberger, Fischer and Brox, 2015) made the encoder-decoder symmetric, concatenating full-resolution encoder features into the decoder at every scale. It was designed for biomedical images with a few dozen training examples, and it remains the default in medical imaging a decade later. Its combination of heavy augmentation and skip connections is unusually data-efficient.

DeepLab (Chen et al., 2017, 2018) attacked resolution loss with atrous — dilated — convolution:

y(i) = sum over k of  x(i + r * k) * w(k)

• r — dilation rate. r = 1 is a standard convolution.

Dilation enlarges the receptive field without downsampling and without adding parameters. Atrous Spatial Pyramid Pooling runs several rates in parallel and concatenates, capturing objects at multiple scales in one layer. DeepLabv3+ adds a light decoder for sharper boundaries.

PSPNet (Zhao et al., 2017) uses pyramid pooling over four grid scales for global context.

Mask R-CNN (He et al., 2017) extends Faster R-CNN with a per-RoI mask branch. Its key technical contribution is RoIAlign: RoIPool quantised region boundaries to the feature grid, misaligning masks by several pixels; RoIAlign uses bilinear sampling at exact locations instead. That single change contributes most of the mask AP gain reported in the paper.

Transformer-based unification. MaskFormer and Mask2Former (Cheng et al., 2021, 2022) recast all three tasks as mask classification. Predict a set of binary masks, each carrying one class label, using a transformer decoder with learned queries. One architecture reaches state of the art on semantic, instance and panoptic benchmarks. This is now the dominant framing.

SAM (Kirillov et al., 2023) is promptable and class-agnostic, trained on SA-1B — 1.1 billion masks over 11 million images. The design decision that makes it usable is asymmetric cost: a heavy ViT image encoder runs once per image, and a lightweight decoder answers each prompt in milliseconds. That asymmetry is also why it is impractical on CPU, where the encoder is the bottleneck.

Losses#

Cross-entropy, averaged over pixels, is the baseline and is dominated by the majority class.

Dice loss optimises overlap directly:

L_dice = 1 - (2 * sum_i (p_i * g_i) + eps) / (sum_i p_i + sum_i g_i + eps)

• p_i — predicted probability for the foreground at pixel i.
• g_i — ground-truth indicator, 0 or 1.
• eps — smoothing constant, preventing division by zero on empty masks.

Dice is scale-invariant with respect to target size, which is exactly what a small-lesion problem needs. It is also noisier to optimise than cross-entropy, which is why the standard practice in medical segmentation is a weighted sum of both.

Lovász-Softmax (Berman et al., 2018) is a convex surrogate for IoU itself, derived from the Lovász extension of a submodular set function. It optimises the evaluation metric more directly than Dice.

Boundary-aware losses — boundary loss, Hausdorff-distance surrogates — target the failure mode that region-overlap losses are least sensitive to. Worth reaching for when the application cares about the edge rather than the area.

Metrics, and what they conceal#

IoU_k = TP_k / (TP_k + FP_k + FN_k)
mIoU  = (1 / K) * sum over k of IoU_k

Two properties are frequently misunderstood.

mIoU is unweighted by class frequency. That is deliberate and it is the reason to prefer it to pixel accuracy. It also means one rare class scoring badly moves the headline number as much as one dominant class, which cuts both ways.

Classes absent from both prediction and ground truth have an undefined IoU. Different codebases handle this differently — some average over present classes only, some substitute zero, some substitute one. This alone produces published numbers that are not comparable. Check the evaluation code, not the paper's table.

For panoptic segmentation, PQ (Kirillov et al., 2019) decomposes usefully:

PQ = ( sum over matched (p,g) of IoU(p,g) ) / ( |TP| + 0.5|FP| + 0.5|FN| )
   = SQ * RQ

• SQ — segmentation quality, the mean IoU over matched segments.
• RQ — recognition quality, the F1 score over segment matching.

The factorisation is the useful part: it separates "did you find the right objects" from "did you outline them well".

Annotation cost, quantified#

This dominates project planning and is under-discussed.

Cordts et al. (2016) reported that Cityscapes fine annotation and quality control required more than 1.5 hours per image on average, for 5,000 images. Coarse polygon annotation for a further 20,000 images was budgeted at roughly 7 minutes each and is measurably lower quality.

The standard responses, in rough order of return on effort:

• Weak supervision from image-level labels, scribbles, points or boxes. Box-supervised instance segmentation (BoxInst, DiscoBox) reaches a large fraction of full-supervision quality.
• Model-assisted annotation. Predict, then have a human correct. SAM is extremely effective here, and this is currently its most common production use.
• Synthetic data. GTA5 and SYNTHIA for driving; domain adaptation closes part of the resulting gap.
• Self-supervised pre-training. DINOv2 features support strong linear segmentation heads with far fewer labels.

Cost#

For an encoder-decoder at input H x W, the decoder operates at high resolution and its cost is often comparable to the encoder's, even with far fewer parameters. Dilated backbones at output stride 8 rather than 32 raise activation memory roughly sixteen-fold at the deepest stage, which is why DeepLab implementations trade output stride against batch size.

At inference, memory scales as O(H * W * K) for the logits alone. At 1024 x 2048 with 19 Cityscapes classes in float32, the logit tensor is about 159 MB per image before anything else. Compute the argmax in chunks, or in float16, when this matters.

Papers#

• Long, J., Shelhamer, E. & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. arXiv:1411.4038
• Ronneberger, O., Fischer, P. & Brox, T. (2015). U-Net. arXiv:1505.04597
• Cordts, M. et al. (2016). The Cityscapes Dataset for Semantic Urban Scene Understanding. arXiv:1604.01685
• He, K. et al. (2017). Mask R-CNN. arXiv:1703.06870
• Zhao, H. et al. (2017). Pyramid Scene Parsing Network. arXiv:1612.01105
• Chen, L.-C. et al. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv:1706.05587
• Chen, L.-C. et al. (2018). Encoder-Decoder with Atrous Separable Convolution (DeepLabv3+). arXiv:1802.02611
• Berman, M. et al. (2018). The Lovász-Softmax Loss. arXiv:1705.08790
• Kirillov, A. et al. (2019). Panoptic Segmentation. arXiv:1801.00868
• Howard, A. et al. (2019). Searching for MobileNetV3. arXiv:1905.02244 — the LR-ASPP head used above.
• Cheng, B. et al. (2022). Masked-attention Mask Transformer for Universal Image Segmentation (Mask2Former). arXiv:2112.01527
• Kirillov, A. et al. (2023). Segment Anything. arXiv:2304.02643
• Isensee, F. et al. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18.

Open problems#

Boundaries. Region-overlap metrics are insensitive to a few pixels of edge error, and human observers are extremely sensitive to it. Boundary-F and trimap evaluation exist and are rarely reported.

Thin and transparent structures. Hair, wires, mesh, glass. Matting — predicting a continuous alpha rather than a hard label — is the right formulation and is a substantially harder learning problem.

Domain shift. A segmentation model trained on one city, one scanner or one camera degrades sharply on another. Test-time adaptation and domain generalisation both help and neither closes the gap.

Open-vocabulary dense prediction. SAM segments anything and names nothing; CLIP names things and localises poorly. Combining them well — open-vocabulary panoptic segmentation — is an active area, and current systems trail closed-set specialists by a wide margin on any fixed taxonomy.

Evaluation under label ambiguity. For medical images, inter-rater IoU between expert annotators is frequently in the 0.7 to 0.9 range. A model reported at 0.85 may be within human disagreement, and a single ground-truth mask cannot express that. nnU-Net's authors make this point repeatedly and it is still routinely ignored.

What to learn next#

• Vision transformers — the backbone behind most current segmentation models.
• Object detection — boxes, IoU and NMS, which instance segmentation builds on.
• Convolutional neural networks — the encoder underneath the small models used here.