Mixup and CutMix

Setup#

pip install torch torchvision numpy

Written and run against torch 2.13.0 (CPU), torchvision 0.28.0, numpy 2.2.6. No downloads.

v2.MixUp and v2.CutMix live in torchvision.transforms.v2 and have their own how-to page in the official torchvision documentation. Neither works on PIL images, bounding boxes, keypoints or masks; both raise a ValueError. They are classification-only transforms.

What the transforms actually return#

import torch
from torchvision.transforms import v2

torch.manual_seed(0)
NUM_CLASSES = 4
imgs = torch.zeros(4, 3, 8, 8)
imgs[0] = 1.0                                     # image 0 is all white, the rest are black
labels = torch.tensor([0, 1, 2, 3])

mix = v2.MixUp(num_classes=NUM_CLASSES, alpha=1.0)
out_img, out_lab = mix(imgs, labels)
print("image shape in/out:", tuple(imgs.shape), "->", tuple(out_img.shape))
print("label shape in/out:", tuple(labels.shape), "->", tuple(out_lab.shape))
print("soft labels:\n", out_lab.round(decimals=3))
print("each row sums to:", out_lab.sum(1))

cut = v2.CutMix(num_classes=NUM_CLASSES, alpha=1.0)
c_img, c_lab = cut(imgs, labels)
print("\nCutMix pixel values present in image 1:", sorted(set(c_img[1].flatten().tolist())))
print("CutMix soft labels:\n", c_lab.round(decimals=3))

image shape in/out: (4, 3, 8, 8) -> (4, 3, 8, 8)
label shape in/out: (4,) -> (4, 4)
soft labels:
 tensor([[0.8640, 0.0000, 0.0000, 0.1360],
        [0.1360, 0.8640, 0.0000, 0.0000],
        [0.0000, 0.1360, 0.8640, 0.0000],
        [0.0000, 0.0000, 0.1360, 0.8640]])
each row sums to: tensor([1., 1., 1., 1.])

CutMix pixel values present in image 1: [0.0, 1.0]
CutMix soft labels:
 tensor([[0.8750, 0.0000, 0.0000, 0.1250],
        [0.1250, 0.8750, 0.0000, 0.0000],
        [0.0000, 0.1250, 0.8750, 0.0000],
        [0.0000, 0.0000, 0.1250, 0.8750]])

Deterministic with the seed set.

Four things that output tells you#

Labels changed shape, from (4,) to (4, 4). Class indices became probability rows. Any code downstream that expects integers now breaks, including your accuracy calculation.

Each image is paired with exactly one other, and it is the batch rolled by one. Row 0 mixes class 0 with class 3, row 1 mixes class 1 with class 0. The implementation is label.roll(1, 0), so consecutive samples get paired. torchvision's own docstring calls this an implementation detail and warns that the batch must be shuffled. With shuffle=False on a sorted dataset, every image is mixed with its neighbour. That neighbour is often the same class, so the transform does nothing. A batch size of 1 mixes an image with itself.

The mixing fraction is shared across the whole batch. Every row uses 0.864 for MixUp, 0.875 for CutMix. One value of lambda is drawn per batch, not per image.

CutMix image 1 contains only 0.0 and 1.0. Image 1 was black and image 0 was white. After CutMix the result holds pure black and pure white pixels, no grey. That is the difference from MixUp in one line: CutMix pastes, it does not blend. Its 0.125 label share is the pasted rectangle's fraction of the total area, which is 8 of 64 pixels.

Wiring it into a DataLoader#

Mixing needs a whole batch, so it belongs in the collate function, not in the per-sample transform.

import torch, torch.nn.functional as F
from torch.utils.data import DataLoader, default_collate, TensorDataset
from torchvision.transforms import v2

torch.manual_seed(0)
NUM_CLASSES = 4
ds = TensorDataset(torch.randn(16, 3, 8, 8), torch.randint(0, NUM_CLASSES, (16,)))
mix = v2.RandomChoice([v2.CutMix(num_classes=NUM_CLASSES), v2.MixUp(num_classes=NUM_CLASSES)])

def collate(batch):                      # mixing happens after samples are batched
    return mix(*default_collate(batch))

dl = DataLoader(ds, batch_size=4, shuffle=False, collate_fn=collate)
xb, yb = next(iter(dl))
print("batch images:", tuple(xb.shape), " batch labels:", tuple(yb.shape), yb.dtype)

logits = torch.randn(4, NUM_CLASSES)
print("cross_entropy with soft targets:", round(F.cross_entropy(logits, yb).item(), 4))
print("hard labels for accuracy:", yb.argmax(1).tolist())

batch images: (4, 3, 8, 8)  batch labels: (4, 4) torch.float32
cross_entropy with soft targets: 1.3216
hard labels for accuracy: [3, 1, 1, 1]

F.cross_entropy accepts probability targets directly, so no custom loss is needed. The PyTorch documentation notes that index targets compute faster. It advises probability targets when one label per item is too restrictive, which is this case. v2.RandomChoice picks one of the two transforms per batch, which is the standard recipe. See custom collate functions.

Does it help? Measure both things#

import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader, default_collate
from torchvision.transforms import v2

rng = np.random.default_rng(0)
NUM_CLASSES = 4

def draw(cls):                                    # noisy shapes: a genuinely hard task
    img = rng.normal(0.4, 0.30, (32, 32))
    r, c = rng.integers(9, 23, 2); s = int(rng.integers(4, 7))
    yy, xx = np.ogrid[:32, :32]; d = np.hypot(yy - r, xx - c)
    m = [(abs(yy-r) < s) & (abs(xx-c) < s), d < s, abs(d-s) < 1.2,
         (abs(yy-r) < 1.2) | (abs(xx-c) < 1.2)][cls]
    img[m] += 0.55
    return np.clip(img, 0, 1).astype("float32")

def make(n):
    lab = rng.integers(0, 4, n)
    return torch.tensor(np.stack([draw(c) for c in lab])).unsqueeze(1), torch.tensor(lab)

Xtr, ytr = make(1500); Xte, yte = make(800)

def ece(probs, y, bins=10):                       # expected calibration error
    conf, pred = probs.max(1)
    correct = (pred == y).float()
    e = 0.0
    for i in range(bins):
        m = (conf > i/bins) & (conf <= (i+1)/bins)
        if m.sum() > 0:
            e += (m.float().mean() * (correct[m].mean() - conf[m].mean()).abs()).item()
    return e

def run(mix):
    torch.manual_seed(0)
    net = nn.Sequential(nn.Conv2d(1,16,3,padding=1), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(2),
                        nn.Conv2d(16,32,3,padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.MaxPool2d(2),
                        nn.Flatten(), nn.Linear(32*8*8, 4))
    opt = torch.optim.AdamW(net.parameters(), lr=1e-3)
    coll = (lambda b: mix(*default_collate(b))) if mix else None
    dl = DataLoader(TensorDataset(Xtr, ytr), batch_size=32, shuffle=True, collate_fn=coll)
    for _ in range(15):
        net.train()
        for xb, yb in dl:
            opt.zero_grad(); F.cross_entropy(net(xb), yb).backward(); opt.step()
    net.eval()
    with torch.no_grad():
        p = F.softmax(net(Xte), 1)
    return (p.argmax(1) == yte).float().mean().item(), p.max(1).values.mean().item(), ece(p, yte)

for name, mix in [("no mixing", None),
                  ("MixUp   ", v2.MixUp(num_classes=NUM_CLASSES, alpha=0.2)),
                  ("CutMix  ", v2.CutMix(num_classes=NUM_CLASSES, alpha=1.0))]:
    a, c, e = run(mix)
    print(f"{name}  accuracy {a:.3f}   mean confidence {c:.3f}   calibration error {e:.3f}")

no mixing  accuracy 0.738   mean confidence 0.854   calibration error 0.116
MixUp     accuracy 0.729   mean confidence 0.779   calibration error 0.055
CutMix    accuracy 0.733   mean confidence 0.705   calibration error 0.057

Deterministic across runs with the seed set.

This is the honest picture#

Accuracy did not improve. 0.738 without mixing, 0.729 and 0.733 with. Under one point, in the wrong direction, on 800 test images. Anyone claiming mixup reliably adds accuracy on a fifteen-epoch run over 1,500 images is selling something.

Calibration error halved, 0.116 to 0.055. The baseline was 73.8 percent accurate while averaging 85.4 percent confidence — an eleven-point overconfidence gap. MixUp pulled average confidence down to 77.9 percent, much closer to the truth.

That is the trade you are making. Say your product thresholds on confidence: auto-approve above 0.95, send the rest to a human. Calibration decides whether that threshold means anything. If you only report top-1 accuracy, mixing will look like wasted effort.

Published accuracy gains come from a different regime. CutMix reports ResNet-50 on ImageNet rising from 76.3 to 78.6 top-1. Those runs were hundreds of epochs over 1.28 million images. That regime is dominated by overfitting; a fifteen-epoch run on noisy shapes is not.

Common mistakes#

Mixing the validation set. Your metric becomes meaningless. Fix: set collate_fn on the training loader only.

Computing accuracy against soft labels. (pred == yb) compares an index against a probability row and raises or silently broadcasts. Fix: yb.argmax(1) when labels are soft, and keep an unmixed copy if you need per-class metrics.

Using it with a five-epoch fine-tune. Mixing makes the training task harder, and a short schedule never recovers the cost. Fix: use it for runs of 100 epochs or more, or leave it out.

Stacking it with heavy label smoothing. Both soften targets. Together they can flatten the signal enough to slow convergence noticeably. Fix: pick one, or reduce the smoothing factor.

CutMix on small objects. A pasted rectangle can cover the whole lesion. The label still claims most of the image is that class. Fix: prefer MixUp when your object of interest is small relative to the frame.

Try it yourself#

Set alpha=1.0 for MixUp instead of 0.2 and re-run. Higher alpha draws mixing fractions nearer 0.5, so images blend more evenly. Watch accuracy and calibration move in opposite directions, and decide which one your product needs.

What to learn next#

• RandAugment and AutoAugment — the other half of a modern training recipe.
• Custom collate functions — where batch-level transforms belong.
• Loss functions — why cross-entropy takes probability targets without complaint.