Semi-supervised learning
When labels are expensive and unlabeled data is everywhere, semi-supervised learning lets the few labels you have spread along the shape of the rest.
- 7 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Semi-supervised learning trains on a few labeled examples plus a mountain of unlabeled ones, letting the labels spread through the mountain.
Picture a wedding hall where you know exactly six guests: three from the bride's side, three from the groom's. Everyone else is a stranger. But strangers are not standing randomly — they cluster with their own people. So you reason outward: the group around a bride's-side uncle is probably bride's side too. Six known names quietly label two hundred.
That reasoning — labels flow along groups — is semi-supervised learning.
Why it exists
In real projects, collecting data is cheap and labeling it is the expensive part. A hospital has a million X-ray images; a radiologist's hour buys labels for a few dozen. A company logs millions of support tickets; someone must read each one to tag it.
Supervised learning uses only the labeled scraps and wastes the mountain. Unsupervised learning uses the mountain but never learns your categories. Semi-supervised learning takes both. The unlabeled mountain reveals the shape of the data: where the clusters and gaps lie. The few labels then name those shapes.
The quiet assumption doing the work: examples in the same cluster tend to share a label. When that holds, a handful of labels goes a very long way. When it fails, the spreading confidently paints the wrong regions.
How it works
before: after label spreading:
? ? ? ? ? B B B G G
? B ? ? ? → B B B G G
? ? ? ? G ? B B B G G G
one B, one G known labels flowed along the clustersA real example you have seen
Google Photos asks you to name a face two or three times. It then tags that person across thousands of photos. Your few answers spread across the unlabeled pile. Speech and vision giants train on oceans of unlabeled audio and images with a sliver of transcribed data, using descendants of exactly this idea.
Remember this
- Labels are expensive; unlabeled data is nearly free — use both.
- The unlabeled mountain reveals shape; the few labels name the shapes.
- It works when same-cluster-same-label holds, and misleads confidently when it does not.
What to learn next
- Supervised learning — the fully-labeled baseline this improves on.
- Clustering — the shape-finding machinery labels flow along.
- Data labelling — why labels cost so much in the first place.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2 on CPU.
Six labels versus six labels plus a mountain
import numpy as np
from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression
from sklearn.semi_supervised import LabelSpreading
# two crescent-shaped classes, 200 points
X, y = make_moons(n_samples=200, noise=0.08, random_state=42)
rng = np.random.default_rng(0)
labeled = rng.choice(len(y), size=6, replace=False) # we can only afford 6 labels
few = LogisticRegression().fit(X[labeled], y[labeled])
print(f"supervised, 6 labels only : {few.score(X, y):.3f}")
y_partial = np.full_like(y, -1) # -1 marks "unlabeled"
y_partial[labeled] = y[labeled]
semi = LabelSpreading(kernel="rbf", gamma=20).fit(X, y_partial)
print(f"semi-supervised, 6 + 194 blank: {semi.score(X, y):.3f}")supervised, 6 labels only : 0.740 semi-supervised, 6 + 194 blank: 0.965
Same six labels. The only change: the second model was allowed to see the unlabeled points, and accuracy jumped from 0.740 to 0.965.
The walkthrough
The -1 convention. scikit-learn's semi-supervised estimators take the full X and a label array where -1 means "unknown". Everything else is a real label. Building y_partial wrong — zeros instead of -1 — silently trains on 194 fake labels of class 0.
Why the supervised model tops out at 0.74. Six points support one straight line, and crescents interlock — no line separates them. The model cannot know the shape exists; it never saw the mountain.
What LabelSpreading does with the mountain. It builds a similarity graph over all 200 points — the rbf kernel scores each pair by closeness, gamma setting how fast that score fades. Known labels then diffuse across strong edges, like dye dropped into flowing water, until the labeling settles. Labels follow the crescents' curves because the graph does.
Checking spread quality. semi.label_distributions_ holds per-point label probabilities. Points deep in a crescent end up near-certain; points in the thin gap stay ambivalent — worth inspecting before trusting the output downstream.
Common mistakes
Evaluating on the points the labels spread to. Our semi.score(X, y) includes the unlabeled points — fair for judging spreading, but for judging generalisation to new data, hold out a fresh test split that the graph never contained.
Assuming the cluster assumption holds. If classes genuinely interleave — fraud hiding inside normal transactions — labels spread to the wrong neighbours and the model is confidently wrong. Semi-supervised methods amplify whatever geometric truth or lie the features hold.
Self-training on its own mistakes. The other classic recipe — train, pseudo-label the confident unlabeled points, retrain (SelfTrainingClassifier) — can snowball an early error into thousands of wrong pseudo-labels. Cap iterations and keep confidence thresholds high.
Scaling neither features nor gamma. The similarity graph is distance-based; unscaled features warp it, and gamma must match the data's scale. Tune gamma — with a labeled validation slice, not by eye.
Try it yourself
Drop noise=0.08 to noise=0.25 so the crescents blur together, and rerun. Watch the semi-supervised advantage shrink as the cluster assumption erodes. Then retry the clean version with size=4 labels — how few can you afford before spreading fails?
What to learn next
- Supervised learning — the fully-labeled baseline this improves on.
- Clustering — the shape-finding machinery labels flow along.
- Data labelling — why labels cost so much in the first place.
Researcher — Mathematics and papers.
Problem setting
Given labeled pairs ${(x_i, y_i)}_{i=1}^{l}$ and unlabeled points ${x_j}_{j=l+1}^{l+u}$ with $u \gg l$, semi-supervised learning seeks a classifier better than one trained on the $l$ pairs alone. Gains require an assumption linking the marginal $p(x)$ — which unlabeled data reveals — to the conditional $p(y \mid x)$: the cluster assumption (decision boundaries avoid high-density regions), its geometric variant the manifold assumption, or consistency under perturbation. No such link, no gain (Chapelle, Schölkopf, Zien, 2006, Semi-Supervised Learning).
Graph-based methods
Build a graph with affinity matrix $W$, $W_{ij} = \exp(-\gamma \lVert x_i - x_j \rVert^2)$, degree matrix $D = \operatorname{diag}(\sum_j W_{ij})$. Label propagation minimises a quadratic energy forcing agreement across strong edges; label spreading (Zhou et al., 2004, Learning with local and global consistency) iterates:
$$ F^{(t+1)} = \alpha \, S F^{(t)} + (1 - \alpha) \, Y $$
Where:
- $F \in \mathbb{R}^{(l+u) \times K}$ — soft label matrix over $K$ classes.
- $S = D^{-1/2} W D^{-1/2}$ — the symmetrically normalised affinity matrix.
- $Y$ — one-hot rows for labeled points, zeros for unlabeled.
- $\alpha \in (0,1)$ — the clamping trade-off: how firmly original labels are held.
The iteration converges to $F^* = (1-\alpha)(I - \alpha S)^{-1} Y$: a closed-form diffusion. Cost is $O((l+u)^3)$ direct or $O(E)$ per sparse iteration with $E$ graph edges; k-NN graphs keep $E$ linear in $n$. The harmonic-function formulation (Zhu, Ghahramani, Lafferty, 2003) clamps labels exactly and interprets $F^*$ via random walks: the probability of hitting each labeled class first.
The wrapper and deep families
- Self-training dates to Scudder (1965); its co-training variant with two feature views is Blum and Mitchell (1998).
- Transductive SVM / S3VM (Joachims, 1999): place the margin through low-density regions — the cluster assumption as a loss term. Non-convex; scalable solvers remain awkward.
- Consistency regularisation powers the deep era: a model's prediction should survive augmentation. Mean Teacher (Tarvainen and Valpola, 2017), MixMatch (Berthelot et al., 2019), FixMatch (Sohn et al., 2020) — pseudo-label confident weak augmentations, enforce agreement on strong ones. FixMatch reaches ~95% on CIFAR-10 with 40 labels, four per class.
- Self-supervised pretraining plus fine-tuning (SimCLR v2; Chen et al., 2020) now often subsumes explicit semi-supervised machinery: the unlabeled mountain shapes representations first, labels arrive last.
The failure mode carries a name worth remembering: under assumption violation, unlabeled data can provably degrade performance (Ben-David et al., 2008) — the theoretical form of "confidently painting the wrong region".
What to learn next
- Supervised learning — the fully-labeled baseline this improves on.
- Clustering — the shape-finding machinery labels flow along.
- Data labelling — why labels cost so much in the first place.