Deploying responsibly

Selective prediction#

Abstention is formalised as a predictor $f$ paired with a selection function $g: \mathcal{X} \to {0, 1}$:

$$ (f,g)(x) = \begin{cases} f(x) & g(x) = 1 \ \text{abstain} & g(x) = 0 \end{cases} $$

Two quantities characterise it:

$$ \text{coverage}(g) = \mathbb{E}[g(x)], \qquad \text{risk}(f,g) = \frac{\mathbb{E}\big[\ell(f(x), y)\,g(x)\big]}{\mathbb{E}[g(x)]} $$

The risk-coverage curve is the object to report; a single operating point hides the trade. El-Yaniv and Wiener (2010) establish the framework, and Geifman and El-Yaniv (2017) give SelectiveNet-style bounds for deep models with a guaranteed risk at a target coverage.

Softmax response is a strong baseline. Better selectors: deep ensembles (Lakshminarayanan et al., 2017), MC dropout, and learned selection heads. Ovadia et al. (2019) benchmark uncertainty methods under distribution shift and find that most degrade sharply — the setting where the selector matters most is the setting where it works least well.

Calibration#

A model is calibrated if $P(Y = \hat{y} \mid \hat{p} = p) = p$. Guo et al. (2017) show modern networks are systematically over-confident, worsening with depth and width, and that single-parameter temperature scaling — dividing logits by $T$ fitted on a validation set by minimising NLL — removes most of the miscalibration without changing the argmax.

Expected calibration error, binned into $M$ bins:

$$ \text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\,\big|\,\text{acc}(B_m) - \text{conf}(B_m)\,\big| $$

ECE is sensitive to binning scheme and bin count; report the scheme, or prefer an adaptive-binning or kernel estimator. Calibration on the training distribution does not transfer under shift, which is the finding most often ignored when a threshold is set once and never revisited.

Conformal prediction (Vovk et al.; Angelopoulos and Bates, 2021) offers distribution-free finite-sample coverage guarantees for prediction sets under exchangeability. Exchangeability fails under drift, and that is precisely when guarantees are wanted; split-conformal with periodic recalibration is the practical compromise.

Detecting failure without labels#

Labels arrive late, so live monitoring is largely unsupervised.

• Input drift. Per-feature two-sample tests (Kolmogorov-Smirnov, chi-squared) with multiple-testing correction; MMD for multivariate. Rabanser et al. (2019) evaluate shift-detection pipelines and find dimensionality reduction followed by univariate testing competitive with more elaborate methods.
• Output drift. Shifts in the predicted-class distribution or the confidence histogram. Cheap, and sensitive in practice.
• Proxy signals. Escalation rate, override rate, complaint volume, downstream conversion. Usually the earliest indicator.

Distinguish covariate shift ($P(X)$ changes, $P(Y \mid X)$ fixed) from concept drift ($P(Y \mid X)$ changes). Only the first is detectable from inputs alone; the second requires labels or a proxy for them, which is why concept drift is typically discovered through a business metric.

Rollout as sequential testing#

A canary is a sequential experiment, and evaluating it repeatedly with a fixed-horizon test inflates the false-positive rate. Two correct approaches: fix the sample size in advance, or use always-valid inference — sequential probability ratio tests, or confidence sequences (Howard et al., 2021) — which permit continuous monitoring at controlled error rates.

Practical sizing: detecting a 1-point difference around 97% accuracy at conventional power needs on the order of thousands of samples per arm, not fifty. Compute the minimum detectable effect before the canary starts, and be prepared for the honest answer that your traffic cannot resolve the difference you care about.

Interference matters for interactive systems. Recommender canaries violate SUTVA through shared item pools and feedback loops; cluster or time-sliced randomisation is often necessary. See A/B testing a recommender.

Feedback loops#

A deployed model changes the data it will next be trained on. Predictive policing models allocate patrols, patrols generate arrest records, records train the next model — Ensign et al. (2018) analyse this as a reinforcing process independent of any underlying crime distribution. Recommenders narrow the item distribution users can respond to, so logged feedback is confounded by exposure.

Mitigations: log the propensity of each exposure to permit inverse-propensity-weighted evaluation, retain a randomised holdout stream that never sees model output, and monitor the diversity of the action distribution over time. Sculley et al. (2015) catalogue these and other hidden costs; the paper remains the best short account of why ML systems accumulate maintenance debt faster than ordinary services.

Incident response#

Borrow directly from site reliability practice, adapted for the fact that an ML failure is usually silent rather than loud.

• Define an SLO on a user-visible quantity, not on model accuracy — escalation rate, dispute rate, time to resolution.
• Error budgets convert "is it good enough" into a spend, and make the rollback decision a policy rather than an argument.
• Blameless postmortems with a mandatory question: what signal would have caught this earlier, and why was it not instrumented?
• Practise the rollback. An untested recovery path is a hypothesis.

Papers#

• Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
• Guo et al., On Calibration of Modern Neural Networks, ICML 2017 — arxiv.org/abs/1706.04599
• Geifman and El-Yaniv, Selective Classification for Deep Neural Networks, NeurIPS 2017 — arxiv.org/abs/1705.08500
• Ovadia et al., Can You Trust Your Model's Uncertainty?, NeurIPS 2019 — arxiv.org/abs/1906.02530
• Rabanser, Günnemann and Lipton, Failing Loudly, NeurIPS 2019 — arxiv.org/abs/1810.11953
• Angelopoulos and Bates, A Gentle Introduction to Conformal Prediction, 2021 — arxiv.org/abs/2107.07511
• Ensign et al., Runaway Feedback Loops in Predictive Policing, FAccT 2018 — arxiv.org/abs/1706.09847

What to learn next#

• Monitoring and model drift — the live signals this lesson depends on.
• Model cards and documentation — recording what you launched and why.
• Why AI safety matters — the start of this section, worth rereading now.