fp32, bf16, fp8 and int4

Anatomy#

An IEEE-style float with sign $s$, exponent field $e$ and mantissa field $m$ (with $p$ mantissa bits and bias $B$) represents, for normal numbers:

$$ x = (-1)^s \cdot 2^{e - B} \cdot \left(1 + \frac{m}{2^{p}}\right) $$

Two derived quantities decide everything:

• Dynamic range $\approx 2^{2^{n_e}}$, set by the exponent width $n_e$.
• Relative precision $\epsilon = 2^{-p}$, set by the mantissa width.

bf16 is fp32 truncated to 16 bits, which is why conversion is a shift and why an fp32 accumulator costs nothing to feed. e4m3fn is "finite": it has no infinities, reclaiming those encodings for values, which is why its max is 448 rather than 240.

Micikevicius et al., 2022 (FP8 Formats for Deep Learning, NVIDIA/Arm/Intel) standardised the e4m3/e5m2 pair and the convention of e4m3 for forward tensors and e5m2 for gradients.

Integer quantisation#

Affine quantisation of a group $G$:

$$ q = \mathrm{clamp}!\left(\left\lfloor \frac{x - z}{s} \right\rceil, q_{\min}, q_{\max}\right), \qquad \hat x = s q + z $$

with scale $s$ and zero point $z$. Symmetric quantisation sets $z = 0$ and $s = \max_{G}|x| / q_{\max}$, which is what the code uses and what most weight quantisers do.

Under a uniform-error model, quantisation noise has variance $s^2/12$, so error scales linearly in the group's dynamic range. With a heavy-tailed weight distribution, $\max_G |x|$ is dominated by the largest element in the group, and error scales with that outlier — which is precisely the 0.81 in the first row of the table.

Effective bits per value, including the scale:

$$ b_{\text{eff}} = b + \frac{b_{\text{scale}}}{g} $$

At $b=4$, $b_{\text{scale}}=16$, $g=32$: 4.5 bits. Papers that quote 4 and use $g=64$ are quoting 4.25.

Non-uniform levels#

Uniform integer levels are a poor match for a bell-shaped weight distribution. NF4 (Dettmers et al., 2023, QLoRA) is information-theoretically optimal for normally distributed data: the 16 levels are the quantiles of a standard normal, so each level receives equal probability mass. Combined with double quantisation — quantising the block scales themselves, saving about 0.37 bits per parameter — QLoRA fine-tunes a 65B model on one 48 GB GPU with no measurable quality loss against 16-bit LoRA.

The 4-bit block formats#

Both encode values as E2M1 (1 sign, 2 exponent, 1 mantissa bits), and differ in the scaling structure:

NVFP4's advantages follow directly from the block-size analysis: smaller blocks track local dynamic range more closely, and an E4M3 scale is finer than a power-of-two one. The cost is 0.25 more bits per value and a two-level scaling hierarchy. Both are native on Blackwell fifth-generation tensor cores; on older hardware they are emulated.

NVIDIA, 2025 (Pretraining Large Language Models with NVFP4) report pretraining a 12B model on 10T tokens in NVFP4 with loss and downstream accuracy comparable to an FP8 baseline, which is a stronger claim than post-training quantisation and was far from a foregone conclusion.

Where the difficulty lives: activation outliers#

Weights are approximately Gaussian and quantise well. Activations do not.

Dettmers et al., 2022 (LLM.int8()) found that beyond roughly 6.7B parameters, emergent outlier features appear in a small number of fixed hidden dimensions, with magnitudes up to 20× the rest, and that zeroing them collapses attention and perplexity. Their solution is a mixed-precision decomposition: the outlier dimensions (about 0.1% of them) stay in fp16, everything else goes to int8, with no quality loss at any tested scale.

Xiao et al., 2023 (SmoothQuant) instead migrate the difficulty: since $Y = (X\,\mathrm{diag}(s)^{-1})\cdot(\mathrm{diag}(s)\,W)$ is mathematically identical, choosing $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$ (with $\alpha \approx 0.5$) divides the burden between activations and weights, making both quantisable to int8 with plain kernels. This same idea, applied to weights only, becomes AWQ.

What to quantise, and what not to#

Consistent findings across the literature:

• Embeddings and the output head are disproportionately sensitive; almost all recipes keep them at 16 bits.
• Normalisation parameters are tiny in count and sensitive; keep them.
• The first and last transformer blocks are more sensitive than the middle.
• Attention is more sensitive than the MLP, so mixed-precision schemes give attention more bits.

Papers and specifications#

• Micikevicius et al., Mixed Precision Training, ICLR 2018 — arxiv.org/abs/1710.03740
• Kalamkar et al., A Study of BFLOAT16 for Deep Learning Training, 2019 — arxiv.org/abs/1905.12322
• Dettmers et al., LLM.int8(), NeurIPS 2022 — arxiv.org/abs/2208.07339
• Micikevicius et al., FP8 Formats for Deep Learning, 2022 — arxiv.org/abs/2209.05433
• Xiao et al., SmoothQuant, ICML 2023 — arxiv.org/abs/2211.10438
• Dettmers et al., QLoRA, NeurIPS 2023 — arxiv.org/abs/2305.14314 (NF4)
• Open Compute Project, OCP Microscaling Formats (MX) Specification v1.0, 2023.
• NVIDIA, Pretraining Large Language Models with NVFP4, 2025 — arxiv.org/abs/2509.25149

What to learn next#

• GPTQ — quantising weights while correcting the error as you go.
• AWQ — deciding which weights matter before you round them.
• Mixed precision training — the training side of the same choices.