Why AI safety matters

The vocabulary, made precise#

"AI safety" covers several distinct research programmes that are often conflated. Keeping them separate makes the literature navigable.

Robustness — worst-case behaviour under perturbation or distribution shift. Formally, bounding $\max_{\delta \in \Delta} \mathcal{L}(f_\theta(x + \delta), y)$ over a perturbation set $\Delta$, rather than the expectation under the training distribution. Here $f_\theta$ is the model with parameters $\theta$, $\mathcal{L}$ the loss, $x$ an input, $y$ its label and $\delta$ an allowed perturbation. Covered in adversarial attacks.

Distributional fairness — constraints relating model behaviour across groups defined by a protected attribute $A$. Covered in fairness metrics.

Privacy — bounding what an adversary can infer about a training record from released model outputs. Covered in differential privacy.

Interpretability — recovering a human-checkable account of why $f_\theta$ produced a given output. Covered in explainability.

Alignment — whether the objective being optimised is the objective the deployer intended. This is the least formalised of the five, and the one where confident claims deserve the most suspicion.

Specification gaming and Goodhart#

The failure that unifies the list is objective misspecification. You optimise a measurable proxy $\tilde{R}$ for an unmeasurable target $R$. Optimisation pressure then drives the system toward the region where $\tilde{R}$ and $R$ diverge most.

Manheim and Garrabrant (2018), Categorizing Variants of Goodhart's Law, decompose this into regressional, extremal, causal and adversarial variants. The taxonomy earns its keep: extremal Goodhart predicts that your proxy degrades precisely at the tail you optimised toward, and that tail is where deployment lives.

Krakovna et al. (2020) maintain a widely cited empirical list of specification-gaming instances in reinforcement learning agents. Read it as an existence proof — reward functions that looked complete to their authors routinely admit degenerate optima.

Risk decomposition#

A useful framing for deployment review, adapted from safety engineering:

$$ \text{Risk} = \sum_{h} P(h) \cdot S(h) \cdot \big(1 - D(h)\big) $$

• $h$ — an identified hazard: one specific way the system produces a bad outcome.
• $P(h)$ — probability of occurrence per unit of exposure.
• $S(h)$ — severity if it occurs, in whatever unit the decision uses.
• $D(h)$ — the detect-and-recover factor: probability the failure is caught and reversed before harm lands.

Two consequences fall out. Hazards you did not enumerate contribute zero to this sum and non-zero to reality, so hazard enumeration dominates the exercise. And $D(h)$ is an engineering variable you fully control — logging, human review, staged rollout, rollback — which is usually far cheaper to raise than $P(h)$ is to lower.

Evaluation is the bottleneck#

The empirical literature repeatedly finds that benchmark performance overstates deployed performance.

• Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet? — rebuilding a test set with the original collection protocol produced accuracy drops of roughly 11 to 14 points across every model tested, with the ranking preserved. Absolute numbers on a reused benchmark are not estimates of deployed accuracy.
• Buolamwini and Gebru (2018), Gender Shades — commercial face classifiers with strong aggregate accuracy showed error rates up to 34.7% on darker-skinned women against 0.8% on lighter-skinned men. Disaggregation, not a new model, revealed it.
• Northcutt et al. (2021), Pervasive Label Errors in Test Sets — an average of 3.4% label errors across ten standard benchmarks, enough to reverse model rankings at realistic noise levels.

The methodological conclusion: a single aggregate metric on a static public benchmark is close to uninformative about deployment risk. Disaggregated evaluation with reported uncertainty, plus a held-out shifted slice, is the minimum defensible protocol.

Where the field is#

Structured hazard analysis for ML is immature relative to aviation or medical devices, but converging. The NIST AI Risk Management Framework (2023) organises practice into Govern, Map, Measure, Manage. ISO/IEC 23894 (2023) covers AI risk management. Neither is a proof technique; both are process scaffolding that makes omissions visible.

Formal verification of neural networks — Katz et al. (2017), Reluplex — gives sound guarantees over small networks and bounded input regions. It does not scale to the models being deployed, and honest practice treats it as a research direction rather than a shipping tool.

Papers#

• Amodei et al., Concrete Problems in AI Safety, 2016 — arxiv.org/abs/1606.06565
• Buolamwini and Gebru, Gender Shades, FAccT 2018
• Recht et al., Do ImageNet Classifiers Generalize to ImageNet?, 2019 — arxiv.org/abs/1902.10811
• Manheim and Garrabrant, Categorizing Variants of Goodhart's Law, 2018 — arxiv.org/abs/1803.04585
• Northcutt et al., Pervasive Label Errors in Test Sets, 2021 — arxiv.org/abs/2103.14749
• Hendrycks et al., Unsolved Problems in ML Safety, 2021 — arxiv.org/abs/2109.13916

What to learn next#

• Bias in datasets — where most real failures are born.
• Fairness metrics — turning "is it fair" into numbers you can test.
• Model evaluation — the measurement toolkit this section builds on.