Latency and throughput

Definitions, stated so they cannot be confused#

For a system processing requests with per-request service time $s$ and batch size $B$:

$$ L_{\text{batch}} = t_{\text{fixed}} + B \cdot t_{\text{per-item}}, \qquad X = \frac{B}{L_{\text{batch}}} = \frac{B}{t_{\text{fixed}} + B\, t_{\text{per-item}}} $$

• $L_{\text{batch}}$ — wall-clock time for one call.
• $X$ — throughput in items per second.
• $t_{\text{fixed}}$ — per-call cost independent of batch size: dispatch, binding, allocation, kernel launch.
• $t_{\text{per-item}}$ — marginal cost of one more item.

As $B \to \infty$, $X \to 1/t_{\text{per-item}}$. The achievable throughput gain from batching is therefore exactly

$$ \frac{X_{\infty}}{X_1} = \frac{t_{\text{fixed}} + t_{\text{per-item}}}{t_{\text{per-item}}} = 1 + \frac{t_{\text{fixed}}}{t_{\text{per-item}}} $$

The two tables in the developer block are this ratio measured. The dense model has $t_{\text{fixed}}/t_{\text{per-item}} \approx 8$, giving the 9x observed. The convolutional model has $t_{\text{fixed}} \approx 0$ relative to its compute, giving 1x. Nothing else needs to be said about when batching helps.

Little's Law#

For any stable queueing system,

$$ L = \lambda W $$

where $L$ is the mean number of items in the system, $\lambda$ the arrival rate, and $W$ the mean time an item spends in it. The law is distribution-free — it assumes only stability.

The consequence people misuse: throughput and latency are linked through concurrency, so quoting one without the other is meaningless. A system reporting 10,000 requests per second at 5 ms latency is running about 50 concurrent requests. If it reports 10,000 per second at 500 ms, it is running 5,000, and something is deeply queued.

On a device $\lambda$ is bounded by the sensor — 30 frames per second from a camera — and concurrency is 1. Little's Law then says $W$ must be under 33 ms or the queue grows without bound. Drop frames deliberately rather than letting a queue build, because a queue converts a throughput deficit into unbounded latency.

Why the tail exists#

Sources of variance, roughly in order of magnitude on a device:

1. DVFS transitions. The governor raises and lowers clock frequency on demand. Ramp-up takes milliseconds, so a burst after idle runs at a low clock.
2. Scheduler preemption. Another process takes the core. On big.LITTLE architectures the scheduler may also migrate the thread to a small core mid-run.
3. Thermal throttling. Covered in power and thermal limits.
4. Memory system. Page faults, TLB misses, cache eviction by another process.
5. Garbage collection or allocator behaviour in the host language.

Dean and Barroso's The Tail at Scale (2013) is about fan-out in distributed systems, but the mechanisms transfer directly: shared resources with occasional long occupancy produce heavy-tailed service times, and the tail dominates perceived quality.

Note that a tail measurement is itself noisy. The p99 of 1000 samples is estimated from about 10 observations, and its own confidence interval is wide. Report the sample count alongside the percentile, or the percentile is not a claim.

Roofline, applied to the two tables#

The convolutional model's flat throughput is a roofline result. Arithmetic intensity for a convolution with $C_{\text{in}}$ input channels, $C_{\text{out}}$ output channels and a $k \times k$ kernel over an $H \times W$ output is

$$ I = \frac{2 H W C_{\text{in}} C_{\text{out}} k^2}{4\left(H W C_{\text{in}} + H W C_{\text{out}} + C_{\text{in}} C_{\text{out}} k^2\right)} $$

with the numerator counting multiply-adds and the denominator counting float32 bytes touched. For typical values this is in the tens to hundreds — well past the machine balance point — so the layer is compute-bound. A compute-bound kernel gains nothing from batching, because batching only reduces memory traffic per item.

A dense layer at batch $B$ has $I \approx B/2$: two operations per weight per item, four bytes per weight, amortised over $B$ items. At $B = 1$ this is 0.5 and firmly memory-bound; the reason batching helps a dense layer at all is that it moves $I$ up the roofline. That the effect saturates by $B \approx 128$ in the measurement is the point where the model returns to being compute-bound.

Measurement methodology#

• Pin the thread count. Do not let the runtime infer it, and never benchmark under a container whose cgroup limit differs from nproc.
• Fix the frequency governor where possible (performance on Linux), or report that you did not.
• Report cold and warm separately. Model load, graph optimisation and first-allocation costs are real user-visible costs and are not latency.
• Use a steady arrival rate, not a closed loop, when the deployment has one. A closed-loop benchmark that issues the next request only after the previous returns cannot produce queueing, and therefore cannot reproduce the tail that queueing causes. This is the coordinated omission problem, named by Gil Tene.
• Report the sample count and the percentile together.

MLPerf Inference (Reddi et al., 2020) codifies this into four scenarios — SingleStream, MultiStream, Server, Offline — precisely because a single number cannot describe both a phone and a datacentre. SingleStream reports 90th-percentile latency; Offline reports throughput. They are different benchmarks of the same model, and neither can be derived from the other.

Reading#

• Dean and Barroso, The Tail at Scale, CACM 2013.
• Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — arxiv.org/abs/1911.02549
• Williams, Waterman and Patterson, Roofline, CACM 2009.
• Gil Tene, How NOT to Measure Latency — the standard talk on coordinated omission.
• Little, A Proof for the Queuing Formula L = λW, Operations Research 1961.

What to learn next#

• Power and thermal limits — why a benchmark's tenth minute differs from its first.
• Model serving — the same trade-off, with the opposite answer.
• Monitoring and model drift — keeping the measurement running after launch.