Frame sampling strategies

A model can only look at a handful of frames, so choosing which handful is a design decision that quietly sets the ceiling on your accuracy.

By Pranay Mahendrakar
12 min read
3 reading levels
Updated 31 Aug 2026

Read these first

On this page 7

Why it exists
The two families
A worked example you can feel
Where you have already seen this
What is honestly hard here
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Frame sampling is deciding which few pictures out of thousands the model gets to look at.

Imagine a friend went to a three-hour wedding and you asked what happened. They cannot replay the whole event for you. They pick eight moments and describe those.

If they pick eight moments from the first ten minutes, you learn about the arrival and nothing else. Spread the eight across the whole evening and you get the shape of the day, minus the details.

That choice — which moments to pick — is frame sampling. A video model faces exactly the same problem.

Why it exists

A ten-second phone video holds around three hundred frames. A one-hour CCTV recording holds over a hundred thousand.

No model reads them all. Memory runs out, and neighbouring frames look almost identical anyway. Feeding a model two nearly identical pictures teaches it very little.

So every video model takes a fixed number of frames. Eight, sixteen and thirty-two are the usual counts. The question is which ones.

The two families

   full video: 300 frames
   |------------------------------------------------|

   DENSE CLIP  (a short window, seen in detail)
   |----XXXXXXXX------------------------------------|
        ^ 8 frames close together

   SPREAD OUT  (the whole video, seen thinly)
   |--X-----X-----X-----X-----X-----X-----X-----X---|
      ^ 8 frames far apart

A dense clip takes frames that sit close together in time. It sees fine detail of a short moment. It cannot see anything outside its window.

A spread-out sample takes frames from across the whole video. It sees the overall story. It may skip straight over a fast event.

Neither is correct in general. The right choice depends on what you are trying to recognise.

A worked example you can feel

Recognising "someone is swimming" needs almost nothing. Any single frame shows water and a person. One well-chosen frame does the job.

Recognising "someone opened a door then closed it again" is different. Open and close look identical unless you see the order. You need frames spread across the whole action.

Recognising a coin being flipped needs frames very close together. The coin is in the air for a fraction of a second.

Where you have already seen this

A video thumbnail on YouTube: one frame chosen to represent the whole video.
The preview strip that appears when you drag the seek bar.
A sports replay slowed down, which is dense sampling made visible.
Phone gallery apps that summarise a long video into a few seconds.

What is honestly hard here

There is no setting that works everywhere, and the failure is silent.

Your model will train, report a number, and quietly be blind to whatever your sampler kept skipping. Nothing crashes. Nothing warns you.

The only defence is to look at what your sampler picks, on real videos, with your own eyes. Read that sentence twice. It is the step almost everybody skips.

Remember this

A model sees a handful of frames, never the whole video.
Dense sampling sees a short moment in detail. Spread-out sampling sees the whole story thinly.
The wrong sampler makes your model blind in a way that produces no error message.

What to learn next

Optical flow — measuring the motion between the frames you sampled.
3D convolutions for video — the first architecture built to read a stack of frames.
Data augmentation — why training and test samplers differ on purpose.

Developer — Code and libraries.

Setup

bash

pip install numpy

Run against numpy 1.26.4 on Python 3.10. Sampling is arithmetic on frame indices, so nothing here depends on a deep learning framework.

Two samplers, and how often they catch an event

sampling.py

import numpy as np

TOTAL = 300          # a 10-second clip at 30 frames per second
NUM = 8              # the model wants exactly 8 frames, no more, no fewer


def dense_clip(total, num, stride=8, rng=None):
    """A contiguous window: num frames, `stride` apart. What I3D and SlowFast use."""
    span = (num - 1) * stride + 1
    start = 0 if rng is None else int(rng.integers(0, max(total - span, 0) + 1))
    return np.arange(start, start + span, stride) % total


def tsn_segments(total, num, rng=None):
    """Split into num equal segments, take one frame from each (Wang et al., 2016)."""
    edges = np.linspace(0, total, num + 1)
    lows, highs = edges[:-1], edges[1:]
    if rng is None:                                   # test time: the middle of each segment
        return (lows + highs) / 2
    return lows + rng.random(num) * (highs - lows)    # train time: random inside each


rng = np.random.default_rng(0)
print("dense clip, stride 8, from the start :", dense_clip(TOTAL, NUM))
print("dense clip, stride 8, random start   :", dense_clip(TOTAL, NUM, rng=rng))
print("TSN segments, centre (test time)     :", tsn_segments(TOTAL, NUM).astype(int))
print("TSN segments, random (train time)    :", tsn_segments(TOTAL, NUM, rng=rng).astype(int))


def hit_rate(sampler, event, trials=4000, **kw):
    """How often does this sampler land on at least one frame of the event?"""
    r = np.random.default_rng(1)
    ev = set(event)
    return sum(bool(ev & set(sampler(TOTAL, NUM, rng=r, **kw).astype(int)))
               for _ in range(trials)) / trials


long_event = range(140, 150)    # 10 frames: a door swinging open
short_event = range(140, 142)   # 2 frames: a hand slapping a table

print(f"\n{'sampler':28s}{'10-frame event':>16s}{'2-frame event':>15s}")
for name, sampler, kw in [("dense, stride 1", dense_clip, dict(stride=1)),
                          ("dense, stride 4", dense_clip, dict(stride=4)),
                          ("dense, stride 8", dense_clip, dict(stride=8)),
                          ("TSN segments", tsn_segments, {})]:
    print(f"{name:28s}{hit_rate(sampler, long_event, **kw):16.3f}"
          f"{hit_rate(sampler, short_event, **kw):15.3f}")

Output

dense clip, stride 8, from the start : [ 0  8 16 24 32 40 48 56]
dense clip, stride 8, random start   : [207 215 223 231 239 247 255 263]
TSN segments, centre (test time)     : [ 18  56  93 131 168 206 243 281]
TSN segments, random (train time)    : [ 10  39  75 142 184 210 252 282]

sampler                       10-frame event  2-frame event
dense, stride 1                        0.060          0.033
dense, stride 4                        0.140          0.056
dense, stride 8                        0.276          0.069
TSN segments                           0.264          0.049

What those numbers are telling you

A stride-1 clip covers 8 frames out of 300. It catches the 10-frame event 6% of the time. Sampling every consecutive frame feels thorough and is the narrowest possible view of a video. Neighbouring frames are nearly identical, so you paid for 8 frames and received roughly one frame's worth of information.

Widening the stride widens the window. Stride 8 spans 57 frames instead of 8, and the hit rate rises from 0.060 to 0.276. Same cost to the model, four times the chance of seeing the thing you care about.

Widening the stride also costs you the short event. Look at the second column. Going from stride 1 to stride 8 raises the 2-frame hit rate to only 0.069, because the sampler steps straight over most short events. Coverage and temporal resolution pull in opposite directions, and no setting gives you both.

TSN segments behave like a wide dense clip, with better guarantees. The hit rate on the long event is 0.264, similar to stride 8. The difference is that TSN always covers the entire video, so it cannot miss the last third the way a random-start clip can.

Test-time TSN is deterministic. With rng=None you get segment centres. This matters: an evaluation number that changes between runs because of the sampler is not a number you can compare against anything.

Training and testing use different samplers, on purpose

At training time, randomness inside each segment is free data augmentation. The same video yields different frame sets each epoch, which regularises the model — the same logic as in data augmentation.

At test time, randomness is noise in your metric. Two options are standard:

Deterministic centres, as above. Cheap and reproducible.
Multi-clip evaluation: sample several clips, average the predictions. Higher accuracy, several times the cost. The mechanics are in action recognition in practice.

Report which one you used. A paper comparing its 10-clip evaluation against someone else's single-clip evaluation is comparing two different things.

Common mistakes

Sampling from a variable frame rate video by index. Index 100 is not "at 3.33 seconds" if the intervals are uneven. Convert to timestamps, or transcode to constant frame rate first.

Randomising the sampler at test time and wondering why the score moves. Fix the seed, or use deterministic centres.

Padding short videos by repeating the last frame. The model learns that stillness at the end predicts a class. Loop the video, or drop clips shorter than your window, and record which you chose.

Sampling frames then augmenting each one independently. Random crops that differ per frame destroy motion. Choose the crop, flip and colour jitter once per clip and apply the same transform to every frame in it.

Assuming more frames always helps. Doubling frames doubles compute. Past a point the extra frames are redundant and the accuracy gain is smaller than the gain from spending the same compute on higher resolution.

Try it yourself

Add a sampler that picks 8 frames at random from anywhere in the video, with no segment structure. Measure its hit rate on both events. Then think about what it does to a model that needs to know the order of what it saw.

What to learn next

Optical flow — measuring the motion between the frames you sampled.
3D convolutions for video — the first architecture built to read a stack of frames.
Data augmentation — why training and test samplers differ on purpose.

Researcher — Mathematics and papers.

The formalism

Let a video have N frames and a model consume T frames. A sampler is a map from N to an ordered index set of size T.

Dense clip sampling with stride s and start t0:

I = { t0 + i·s  :  i = 0 … T-1 },   t0 ~ Uniform(0, N - (T-1)s)

The temporal receptive field is (T-1)·s + 1 frames. At 30 frames per second, the widely used 16 x 4 setting covers 61 frames, or roughly 2 seconds. Any action longer than that is, by construction, outside a single clip's view.

Segment sampling (Wang et al., 2016, Temporal Segment Networks) partitions [0, N) into T equal segments and draws one index per segment. Coverage of the full video is guaranteed for every sample, and the variance across draws is bounded by segment width N/T.

The distinction is not cosmetic. Dense clip sampling makes the model a local motion classifier evaluated many times; segment sampling makes it a global aggregator evaluated once. Architectures were designed around one or the other, and swapping the sampler without adjusting the architecture usually loses accuracy.

Which datasets reward which sampler

Dataset	Dominant cue	Sampler that wins
Kinetics-400/600/700	Scene and object context	Either; even single frames score well
UCF-101, HMDB-51	Appearance, small and easy	Either
Something-Something v2	Temporal order, direction	Segment or long-window; single frames fail
AVA	Spatio-temporal localisation	Dense clip centred on the annotated keyframe
Charades	Long, overlapping actions	Multi-clip covering the full video

Something-Something is the useful stress test. Its classes are constructed as pairs like "pushing something left to right" and "pushing something right to left", so any model that discards temporal order is capped near chance on those pairs. Reported accuracy on Kinetics alone tells you almost nothing about temporal modelling, a point made repeatedly since Zhou et al. (2018), Temporal Relational Reasoning.

Test-time aggregation protocols

The literature's standard notation is K x C: K temporal clips, C spatial crops, giving K·C forward passes averaged.

1 x 1: single centre clip, centre crop. Cheapest, lowest accuracy, and the number most often quoted for deployment.
10 x 3: the SlowFast and Video Swin convention. 30 forward passes per video.
4 x 3: common for transformer models where each pass is expensive.

The gap between 1 x 1 and 10 x 3 on Kinetics-400 is typically 2 to 4 points of top-1 accuracy. That is larger than the gap between many published architectures, which is why protocol must be stated whenever accuracies are compared.

Sampling as an implicit prior

Two consequences that are easy to miss:

Frame rate normalisation is a modelling choice. Sampling at fixed frames rather than fixed seconds means the same action at 24 and 60 frames per second produces clips of different real duration. Models trained this way become sensitive to source frame rate, and the sensitivity shows up as a domain gap between datasets.

Stride is a low-pass filter. Sampling at stride s aliases any motion whose period is shorter than 2s frames, by the Nyquist argument. Fast periodic actions — clapping, running, drumming — can alias to a slower apparent motion. This is a genuine, measurable failure mode and one motivation for two-rate architectures such as SlowFast networks.

Learned and adaptive sampling

Fixed samplers waste capacity on uninformative frames. Several lines of work make the choice adaptive:

SCSampler (Korbar et al., 2019) trains a cheap saliency model to rank clips, then runs the expensive classifier on the top few. Large accuracy-per-FLOP gains on long untrimmed video.
AdaFrame (Wu et al., 2019) uses a recurrent policy to decide, frame by frame, whether to look further.
AR-Net (Meng et al., 2020) selects per-frame resolution rather than per-frame inclusion.

All of them add a training-time dependency and a second model to maintain. On trimmed short-clip benchmarks the gain over segment sampling is modest; on untrimmed hour-scale video it is substantial, which is where the method belongs.

References

Wang et al., Temporal Segment Networks: Towards Good Practices for Deep Action Recognition, ECCV 2016 — arxiv.org/abs/1608.00859
Carreira and Zisserman, Quo Vadis, Action Recognition? (I3D and Kinetics), CVPR 2017 — arxiv.org/abs/1705.07750
Goyal et al., The "Something Something" Video Database, ICCV 2017 — arxiv.org/abs/1706.04261
Zhou et al., Temporal Relational Reasoning in Videos, ECCV 2018 — arxiv.org/abs/1711.08496
Korbar et al., SCSampler, ICCV 2019 — arxiv.org/abs/1904.04289

What to learn next

Optical flow — measuring the motion between the frames you sampled.
3D convolutions for video — the first architecture built to read a stack of frames.
Data augmentation — why training and test samplers differ on purpose.

🟢 Beginner — No maths. Plain English.

Why it exists#

The two families#

A worked example you can feel#

Where you have already seen this#

What is honestly hard here#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Two samplers, and how often they catch an event#

What those numbers are telling you#

Training and testing use different samplers, on purpose#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The formalism#

Which datasets reward which sampler#

Test-time aggregation protocols#

Sampling as an implicit prior#

Learned and adaptive sampling#

References#

What to learn next#

Beginner — No maths. Plain English.

Why it exists

The two families

A worked example you can feel

Where you have already seen this

What is honestly hard here

Remember this

What to learn next

Developer — Code and libraries.

Setup

Two samplers, and how often they catch an event

What those numbers are telling you

Training and testing use different samplers, on purpose

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The formalism

Which datasets reward which sampler

Test-time aggregation protocols

Sampling as an implicit prior

Learned and adaptive sampling

References

What to learn next