Learn AI Create the Future — with Pranay
Main site

Segmentation in Depth

Weakly supervised segmentation

Outlining every object by hand is the real cost of segmentation, so weak supervision learns masks from image labels, points, scribbles or boxes instead.

Read these first

On this page 9
  1. The short answer
  2. The analogy
  3. Why it exists
  4. The four cheap labels
  5. How the classic method works
  6. The step that rescues it
  7. What is honestly hard here
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Weak supervision learns to outline objects from labels that never contain an outline.

The analogy

Show a friend a thousand photos. Say only "there is a cat here" or "there is no cat". You never point at the cat.

After a while they get very good at answering the yes-or-no question. Now hand them a pen and ask them to draw around the cat.

They will draw a rough blob over the face. The face is what they were using to decide, so the face is what they learned to find.

That gap — good at deciding, rough at drawing — is the whole subject of this lesson.

Why it exists

Outlining is brutally slow. A careful mask for one street photo takes a skilled annotator several minutes. A tick saying "contains a car" takes a second.

The arithmetic is unforgiving. A hundred thousand outlines is a year of somebody's life. A hundred thousand ticks is a fortnight.

So the field asked a sharp question. How much of the outline can be recovered from labels that are cheap to collect?

The four cheap labels

   cheapest                                        most useful
      |                                                  |
   image tag ---> a single point ---> a scribble ---> a box
   "has a car"    one click on it     a rough line     a rectangle
                                      through it       around it
      |                                                  |
   seconds                                        tens of seconds

Each step up costs more and gives the model more. A box, for instance, tells you exactly where the object is not, which turns out to be enormously informative.

How the classic method works

Train an ordinary yes-or-no classifier on the image tags. Then ask a question it was never trained to answer. Which parts of the picture made you say yes?

The answer comes out as a heat map, brightest where the evidence was strongest. Threshold the heat map and you have a rough mask.

The technique has a name, class activation mapping, usually shortened to CAM. It works, and it fails in two predictable ways.

The map is blurry. It comes from deep inside the network, where the picture has been shrunk many times. And it covers only the convincing part, not the whole object. A cat's face, not the cat.

The step that rescues it

Here is the part that surprises people the first time.

Take those rough, blurry masks and use them to train a second model, as if they were real. The second model ends up better than the masks it learned from.

Why? The rough masks are wrong in scattered, inconsistent ways, and consistent in the parts that matter. A model trained on many of them learns the pattern that repeats and averages away the noise.

That is not a trick. It is the backbone of most weakly supervised systems in use today.

What is honestly hard here

The rough mask needs a cut-off: how bright must the heat be to count as inside the object? That single number changes the result enormously.

To choose it well, you need some real outlines to check against. That is the thing you were trying to avoid.

Most honest papers admit this. They tune on a small set of real masks and report it. Anyone who claims a fully mask-free pipeline with a well-chosen threshold has usually hidden this step.

Remember this

  • Cheap labels: tag, point, scribble, box, in rising order of cost and value.
  • A classifier's heat map gives a rough, blurry mask covering only the obvious part.
  • Training a second model on those rough masks beats the rough masks.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch

CPU only, roughly twenty-five seconds. The synthetic object has a dim body with a very bright core. The classifier can succeed by looking at the core alone, which is the failure mode this field exists to fix.

The whole pipeline, measured

weak_supervision.py
import torch, torch.nn as nn, torch.nn.functional as F

S = 32
def batch(n, gen):
    """Half the images contain a dim 9x9 block with a very bright 3x3 core."""
    x = torch.rand(n, 1, S, S, generator=gen) * 0.35
    y = torch.zeros(n, 1, S, S)                      # the true mask, used ONLY for scoring
    label = torch.zeros(n, dtype=torch.long)
    for i in range(n):
        if torch.rand(1, generator=gen).item() < 0.5:
            continue
        label[i] = 1
        r, c = torch.randint(3, S - 12, (2,), generator=gen).tolist()
        x[i, 0, r:r+9, c:c+9] += 0.15                # the dim body
        x[i, 0, r+3:r+6, c+3:c+6] += 0.60            # the bright core
        y[i, 0, r:r+9, c:c+9] = 1.0
    return x.clamp(0, 1), y, label

class CamNet(nn.Module):
    """Conv stack -> global average pool -> one linear layer. The CAM recipe."""
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(32, 32, 3, padding=1), nn.ReLU())
        self.classifier = nn.Linear(32, 2)
    def forward(self, x):
        f = self.features(x)
        return self.classifier(f.mean((-2, -1))), f
    def cam(self, x, cls=1):
        _, f = self(x)
        w = self.classifier.weight[cls].view(1, -1, 1, 1)
        m = (w * f).sum(1, keepdim=True)                      # weighted sum of feature maps
        m = F.interpolate(m, size=(S, S), mode="bilinear", align_corners=False)
        m = m - m.amin((-2, -1), keepdim=True)
        return m / m.amax((-2, -1), keepdim=True).clamp(min=1e-6)

def iou(pred, gt):
    keep = gt.sum((1,2,3)) > 0
    inter = (pred * gt).sum((1,2,3))[keep]
    union = ((pred + gt) > 0).float().sum((1,2,3))[keep]
    return (inter / union).mean().item()

vg = torch.Generator().manual_seed(5)
xv, yv, lv = batch(200, vg)

# ---------- trained on image-level labels only ----------
torch.manual_seed(0)
weak = CamNet(); opt = torch.optim.Adam(weak.parameters(), lr=3e-3)
g = torch.Generator().manual_seed(1)
for _ in range(600):
    x, _, lab = batch(16, g)
    logits, _ = weak(x)
    loss = F.cross_entropy(logits, lab)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    acc = (weak(xv)[0].argmax(1) == lv).float().mean().item()
    cam = weak.cam(xv)
print(f"classifier accuracy on the only labels it ever saw: {acc:.1%}")
print(f"\n{'CAM threshold':>15}{'IoU with the true block':>26}")
for t in (0.2, 0.3, 0.5, 0.7):
    print(f"{t:>15.1f}{iou((cam > t).float(), yv):>26.3f}")

# ---------- the same data, trained on real masks ----------
class SegNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(nn.Conv2d(1,16,3,padding=1), nn.ReLU(),
                                  nn.Conv2d(16,16,3,padding=1), nn.ReLU(),
                                  nn.Conv2d(16,1,1))
    def forward(self, x): return self.body(x)

torch.manual_seed(0)
strong = SegNet(); opt = torch.optim.Adam(strong.parameters(), lr=3e-3)
g = torch.Generator().manual_seed(1)
for _ in range(600):
    x, y, _ = batch(16, g)
    loss = F.binary_cross_entropy_with_logits(strong(x), y)
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    print(f"\nsame data, but trained on real masks: IoU {iou((strong(xv) > 0).float(), yv):.3f}")

# ---------- the standard pipeline: train a segmenter on CAM pseudo-labels ----------
torch.manual_seed(0)
student = SegNet(); opt = torch.optim.Adam(student.parameters(), lr=3e-3)
g = torch.Generator().manual_seed(1)
for _ in range(600):
    x, _, lab = batch(16, g)
    with torch.no_grad():
        pseudo = (weak.cam(x) > 0.5).float() * lab.view(-1, 1, 1, 1).float()
    loss = F.binary_cross_entropy_with_logits(student(x), pseudo)   # never sees a real mask
    opt.zero_grad(); loss.backward(); opt.step()
with torch.no_grad():
    print()
    print(f"CAM pseudo-labels at 0.5, scored against true masks: IoU {iou((weak.cam(xv) > 0.5).float(), yv):.3f}")
    print(f"a student trained on those pseudo-labels:            IoU {iou((student(xv) > 0).float(), yv):.3f}")

# ---------- look at one example ----------
i = int(lv.nonzero()[0])
print("\ntrue block (#) next to the CAM above 0.3 (o), rows 4-19:")
for r in range(4, 20):
    a = "".join("#" if v else "." for v in yv[i,0,r])
    b = "".join("o" if v > 0.3 else "." for v in cam[i,0,r])
    print(f"   {a}   {b}")
Output
classifier accuracy on the only labels it ever saw: 100.0%

  CAM threshold   IoU with the true block
            0.2                     0.277
            0.3                     0.354
            0.5                     0.591
            0.7                     0.827

same data, but trained on real masks: IoU 0.862

CAM pseudo-labels at 0.5, scored against true masks: IoU 0.591
a student trained on those pseudo-labels:            IoU 0.718

true block (#) next to the CAM above 0.3 (o), rows 4-19:
   ................................   ......ooooooooooooo.............
   ................................   ......oooooooooooooo............
   ........#########...............   .....ooooooooooooooo............
   ........#########...............   .....ooooooooooooooo............
   ........#########...............   .....oooooooooooooooo...........
   ........#########...............   ....ooooooooooooooooo...........
   ........#########...............   ....ooooooooooooooooo...........
   ........#########...............   ....ooooooooooooooooo...........
   ........#########...............   .....ooooooooooooooo............
   ........#########...............   .....ooooooooooooooo............
   ........#########...............   .....ooooooooooooooo............
   ................................   ......ooooooooooooo.............
   ................................   .......ooooooooooo..............
   ................................   .........ooooooo................
   ................................   ................................
   ................................   ................................

Seeded, on PyTorch 2.5.1 running on CPU. Expect small variation across versions.

Reading the output

The classifier is perfect, and that tells you nothing about the mask. One hundred per cent on the yes-or-no question, from an object whose bright core is easy to spot. Classification accuracy is not evidence of localisation quality, and papers that report only the former should be read carefully.

The threshold sweep is the honest headline. The same heat map yields IoU from 0.277 to 0.827 depending on one number. Nothing about the model changed between those rows.

That is the catch stated at the start. Picking 0.7 rather than 0.2 needs real masks to compare against. A pipeline that tunes its threshold on real masks is not fully weakly supervised. The honest thing is to say how many masks were used.

The printed comparison shows the shape problem. The true block is a crisp 9-by-9 square. The CAM is a soft, rounded, oversized blob drifting a couple of pixels left. It comes from a feature map at one quarter resolution, then bilinearly stretched. No threshold recovers a square corner from that.

The student beat its teacher: 0.718 against 0.591. It never saw a real mask. It was trained purely on the CAM masks and scored 12.7 points above them.

The reason is worth stating precisely. The pseudo-labels are wrong in ways that vary from image to image — the blob drifts in different directions, spreads by different amounts. The bright block itself is always in the same place relative to the image evidence. Averaged over many images, the inconsistent errors cancel and the consistent signal survives.

The fully supervised ceiling here is 0.862. So the weak pipeline recovers roughly 83 per cent of the achievable score, using zero mask annotations. That ratio is in the same range that published methods report on real datasets, which is a reasonable sanity check on this toy.

Making it stronger, in the order that pays

  1. Higher-resolution CAMs. Use a backbone with output stride 8 or 16 rather than 32, or run at a larger input size. This is usually the cheapest large win.
  2. Better seeds. Grad-CAM works on any architecture without the global-average-pool constraint. Multi-scale and flipped averaging reduces variance for free.
  3. Expansion. CAMs cover the discriminative region only. Affinity-based expansion, such as AffinityNet or IRN, grows the seed along learned pixel-similarity, filling in the dim body.
  4. A stronger label. A box is far more informative than a tag, and boxes are still cheap. Box-supervised methods reach a large fraction of fully supervised quality.
  5. Prompt a foundation model. Feed your boxes or points to a promptable model and take its masks as pseudo-labels. See Segment Anything. This is the modern default and it often beats a decade of CAM refinement.

Common mistakes

Reporting the pseudo-label quality as the method's result. Report the student, evaluated on a held-out set with real masks.

Tuning the threshold on the test set. The threshold is a hyperparameter. Tune it on validation data and say so.

Using the CAM of the predicted class rather than the true class. During pseudo-label generation you know the image label. Use it.

Forgetting to zero out negative images. The * lab.view(-1, 1, 1, 1) in the script matters. Without it, every background image contributes a confident, hallucinated mask.

Assuming this transfers to instance segmentation. CAMs merge touching objects of one class. Separating instances from image tags alone is much harder, and needs point or box supervision in practice.

Try it yourself

Change the dim body from += 0.15 to += 0.05, making it nearly invisible. The classifier still scores near perfect from the core alone, and the CAM shrinks towards the core. That is the discriminative-region problem in a controlled setting.

What to learn next

Researcher — Mathematics and papers.

The supervision ladder

Ordered by annotation cost, with the constraint each label supplies:

SupervisionConstraint providedRepresentative work
Image tagclass is present somewhereCAM (Zhou et al., 2016), SEC, AffinityNet, IRN
Pointone pixel is definitely foregroundBearman et al., 2016, What's the Point?
Scribblea connected set of foreground pixelsScribbleSup (Lin et al., 2016)
Boxtight extent; everything outside is backgroundBoxSup (Dai et al., 2015), BoxInst (Tian et al., 2021)
Full maskexact—

Bearman et al. (2016) made the argument that reframed the field. Measure against a fixed annotation budget rather than a fixed number of images. Point supervision then beat full masks, because the same money buys far more images.

That framing is the right one for practice. The question is not "how close to full supervision?" but "what is the best mask quality per rupee of annotation?".

Why CAMs cover only discriminative regions

For a network with global average pooling followed by a linear classifier, the score for class $c$ is

$$ s_c = \sum_k w_k^c \cdot \frac{1}{Z}\sum_{x,y} f_k(x,y) = \frac{1}{Z}\sum_{x,y} \underbrace{\sum_k w_k^c f_k(x,y)}_{M_c(x,y)} $$

so the class score is exactly the spatial average of the map $M_c$. Zhou et al. (2016) note the identity; the failure mode follows from it.

The classification objective is satisfied once $s_c$ exceeds the decision boundary. Nothing rewards spatial completeness. Once the most discriminative region drives $s_c$ high enough, gradients on the remaining object region vanish. The classifier has no incentive to be a segmenter, so it is not one.

Countermeasures follow directly from the diagnosis. Adversarial erasing removes the peak region and forces the network to find more. Sub-category and equivariance constraints (SEAM) penalise inconsistency across transformations. Affinity propagation (AffinityNet, IRN) learns pixel similarity from confident seeds, then diffuses the labels.

Why the student beats the teacher

The developer result — 0.718 from a student trained on 0.591 pseudo-labels — is the empirical core of nearly every self-training pipeline. The mechanism has three parts.

Noise cancellation under an inductive bias. Suppose pseudo-label errors are approximately independent given the image evidence. If the student's hypothesis class cannot represent that noise, the fitted function approaches the noise-free target. Convolutional weight sharing is doing that work here.

Capacity mismatch. The student is deliberately not powerful enough to memorise the noise. Increase its capacity far enough and it will fit the pseudo-labels exactly, including their errors, and the gain disappears. This is testable and worth testing in your own pipeline.

Consistency regularisation. In real pipelines the student sees augmented views. It is penalised for disagreeing with itself under transformation, which suppresses unstable errors.

The failure case is confirmation bias: errors that are systematic rather than random survive and amplify across self-training rounds. In this toy, the CAM's leftward drift is partly systematic, which is one reason the student stops at 0.718 rather than approaching 0.862.

Box supervision as a projection constraint

BoxInst (Tian et al., 2021) is the cleanest formulation. It drops mask supervision entirely and trains on two terms:

  • Projection loss. The horizontal and vertical projections of the predicted mask must match those of the box. If the predicted mask is $m$ and the box is $b$, the loss compares $\max_y m(x,y)$ against $\max_y b(x,y)$ per column, and likewise per row.
  • Pairwise affinity loss. Neighbouring pixels of similar colour are encouraged to share a label, a learned stand-in for the pairwise term of a conditional random field.

The result reaches a large fraction of fully supervised mask AP with no mask annotation at all. The lesson generalises. A weak label is best used as an explicit constraint on the output, rather than converted into a fake mask.

Where this stands now

Promptable foundation models changed the economics. A box costs seconds to draw, and a promptable model converts a box into a mask that is often close to human quality. Many teams that would have run a CAM pipeline in 2021 now draw boxes and generate masks. They spot-check those, then train a small deployable model on them.

Two cautions. Foundation-model masks inherit that model's biases, so systematic errors on your domain propagate into every downstream model you train. Zero-shot quality on specialised imagery is often much lower than natural-image benchmarks suggest. Histopathology, radar and industrial inspection all need measuring on your own data.

Papers

What to learn next