Scaling laws and compute-optimal training

The parametric form#

$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} $$

$N$ is non-embedding parameter count, $D$ training tokens. $E$ is the irreducible loss — the entropy of natural text under this tokeniser, which no model can beat. $A/N^\alpha$ is the penalty for finite capacity; $B/D^\beta$ the penalty for finite data. The additive separable form is an assumption, not a derivation, and it is the form's main weakness.

Kaplan et al., 2020 (Scaling Laws for Neural Language Models) established the power-law behaviour and concluded model size should grow far faster than data: $N \propto C^{0.73}$, $D \propto C^{0.27}$.

Hoffmann et al., 2022 (Training Compute-Optimal Large Language Models) corrected this with over 400 runs and three independent methods, finding $N \propto C^{0.50}$, $D \propto C^{0.50}$. The discrepancy was traced largely to Kaplan et al. using a fixed cosine schedule length across all token counts, and to counting embedding parameters — both of which bias against long training runs.

Under the Hoffmann analysis the optimal allocation satisfies

$$ N_{\text{opt}} \propto C^{a},\quad D_{\text{opt}} \propto C^{b},\quad a = \frac{\beta}{\alpha+\beta},\quad b = \frac{\alpha}{\alpha+\beta} $$

so the tokens-per-parameter ratio is constant in $C$ only when $\alpha = \beta$. This is the crux of the replication dispute below.

The replication problem, in numbers#

Besiroglu et al., 2024 (Chinchilla Scaling: A replication attempt, Epoch AI) reconstructed the data from Hoffmann's Figure 4 and refitted Approach 3:

Their three findings: Hoffmann's Approach-3 estimates are inconsistent with its own Approaches 1 and 2; they fit the reconstructed data poorly; and the reported confidence intervals are implausibly narrow — tight enough to require over 600,000 runs against roughly 400 actually performed. They attribute the discrepancy to an optimiser stopping short of convergence and to rounding in the published coefficients.

The practical takeaway: use the Besiroglu coefficients if you want a fit consistent with the paper's own headline conclusion, and treat any downstream theory built on the original Approach-3 exponents with care.

Overtraining, and the inference-aware law#

Compute-optimal minimises training FLOPs for a target loss. Sardana et al., 2024 (Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws) minimise training plus lifetime inference instead. With $D_{\text{inf}}$ tokens expected to be generated over the model's life, the objective becomes

$$ \min_{N, D_{\text{tr}}} \; 6 N D_{\text{tr}} + 2 N D_{\text{inf}} \quad \text{s.t.} \quad L(N, D_{\text{tr}}) = \ell $$

The $2N$ per inference token is the forward-pass cost. For realistic serving volumes the solution is substantially smaller and substantially more overtrained than Chinchilla — matching what Llama 3 8B (15T tokens, ~1900 tokens/param) and similar models actually do.

Distillation shifts the same trade further, and knowledge distillation scaling laws (Busbridge et al., 2025) give the analogous frontier for teacher–student pairs. See distilling a large model into a small one.

Data-constrained scaling#

The additive form implies unlimited benefit from unlimited data. Data is finite. Muennighoff et al., 2023 (Scaling Data-Constrained Language Models) trained on repeated data up to 900B tokens and found:

• Up to roughly 4 epochs, repeated tokens are nearly as valuable as fresh ones.
• Beyond that, the marginal value decays quickly toward zero.
• Their extension replaces $D$ with an effective token count $D'$ that saturates in the number of repetitions.

This is the standard reference for anyone whose corpus is smaller than their compute budget.

Where the framework is weakest#

• Fitted to one setup. One architecture family, one tokeniser, one optimiser, one data mixture. Changing the mixture changes $E$ directly.
• Data quality is unmodelled. Bi et al., 2024 (DeepSeek LLM) show the optimal allocation shifts toward smaller models as data quality rises. There is no quality term in the equation.
• Loss is not capability. The law predicts $L$; downstream accuracy is a separate, noisier mapping. See Schaeffer et al., 2023 on the metric artefacts in that mapping.
• MoE breaks the parameter count. For a mixture-of-experts model, active parameters and total parameters differ, and $6ND$ needs the active count while capacity tracks the total. Separate scaling laws exist for MoE and the dense law does not transfer.
• The optimum is flat. As the code showed, a 2× error costs under 0.01 nats. Precision in these coefficients is worth far less than it appears.

Papers#

• Kaplan et al., Scaling Laws for Neural Language Models, 2020 — arxiv.org/abs/2001.08361
• Hoffmann et al., Training Compute-Optimal Large Language Models, 2022 — arxiv.org/abs/2203.15556
• Muennighoff et al., Scaling Data-Constrained Language Models, NeurIPS 2023 — arxiv.org/abs/2305.16264
• Sardana et al., Beyond Chinchilla-Optimal, 2024 — arxiv.org/abs/2401.00448
• Bi et al., DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, 2024 — arxiv.org/abs/2401.02954
• Besiroglu et al., Chinchilla Scaling: A replication attempt, 2024 — arxiv.org/abs/2404.10102
• Hägele et al., Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, 2024 — arxiv.org/abs/2405.18392

What to learn next#

• Building a pretraining corpus — where the D in these equations comes from.
• Model size versus accuracy — the same trade-off at a scale you can actually run.
• Distilling a large model into a small one — the other way to get a cheap model.