Champion and challenger models
A champion model keeps serving every real decision while one or more challengers are scored continuously alongside it, and only take over after a sustained, statistically real advantage — the same discipline as a table-tennis champion who keeps the table only while they keep winning, not after one lucky point.
- 11 min read
- 3 reading levels
- Published
Read these first
On this page 7
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
A champion model handles every real decision, while one or more challengers are scored alongside it continuously. A challenger only takes over after a sustained, real advantage.
The analogy you have already lived
You have seen the "winner stays on" rule at a table-tennis or carrom table. The reigning champion keeps playing challenger after challenger, and keeps the table only for as long as they keep winning. Nobody hands the table to a challenger after one lucky point in a single rally. A real win means beating the champion decisively, not once, but consistently enough that it was not only luck.
Champion/challenger deployment runs that same rule for models. One model — the champion — makes every real decision. Challengers are evaluated continuously, on the same real requests. They only take the champion's place once they have won decisively enough, for long enough, to trust it.
Why it exists
A canary release is usually about one specific new version, released once, ramped up, and finished. Champion/challenger is a longer-running habit instead: an ongoing way of deciding what deserves to become the next release at all. Several challengers can be evaluated at once, over weeks rather than days.
The real risk it guards against is promoting a challenger too early, on too little evidence. A challenger that looks better after fifty requests can easily be a lucky streak, not a genuine improvement. Champion/challenger builds the patience for that distinction directly into the process, instead of relying on someone's gut feeling about when "enough" evidence has arrived.
How it works
every real request
|
+----------------------> CHAMPION -> this is the answer the user gets
|
+----------------------> CHALLENGER 1 \
+----------------------> CHALLENGER 2 } scored, never shown, like shadow deployment
/
once real outcomes are known (a label, a result):
score champion and each challenger against what actually happened
|
v
has a challenger won decisively AND for long enough to trust it?
/ \
no yes
| |
keep watching PROMOTE: it becomes
the new championA real example you have seen
A credit-scoring team rarely swaps its production model on a single good test result. A newer model is far more often run this way for weeks. It is scored against the same real applications the current model is already deciding on. Its accuracy is tracked over enough real cases that a genuine improvement can be told apart from ordinary week-to-week noise.
Remember this
- The champion serves real traffic; challengers are scored, never shown, exactly like shadow deployment.
- A challenger is promoted only after a sustained, statistically real advantage — not the first sign of one.
- This is an ongoing habit, often running several challengers over weeks, not a one-time release event.
What to learn next
- Swapping weights with no downtime — the mechanics of actually installing a newly promoted challenger as champion.
- Versioning a model API — keeping the promoted model's interface stable for whatever called the previous champion.
- Peeking and sequential testing — the statistically rigorous version of the repeated-checking pattern this arena uses in simplified form.
Developer — Code and libraries.
Setup
pip install scikit-learn pandas numpy scipy pytestThe arena
Every request is scored by both the champion and a challenger. Real outcomes accumulate into running accuracy for each, and a promotion decision reuses the same two-proportion significance test from canary releases for models — this time comparing accuracy over time, instead of comparing error rates between two live traffic slices.
"""A champion/challenger arena: one champion serves real decisions, one or
more challengers are scored continuously alongside it, and a challenger is
only promoted after a sustained, statistically real advantage.
"""
import numpy as np
import pandas as pd
from scipy import stats
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
COLUMNS = ["income", "years", "age"]
def make_stream(n, seed):
rng = np.random.RandomState(seed)
X = pd.DataFrame({
"income": rng.uniform(5, 80, n).round(1),
"years": rng.uniform(0, 10, n).round(1),
"age": rng.randint(21, 65, n).astype(float),
})
score = 0.05 * X["income"] + 0.35 * X["years"] + 0.01 * X["age"] - 3.5
y = (score + rng.normal(0, 0.8, n) > 0).astype(int)
return X, y
class ArenaStats:
def __init__(self, name):
self.name = name
self.correct = 0
self.total = 0
def record(self, predicted, actual):
self.total += 1
self.correct += int(predicted == actual)
@property
def accuracy(self):
return self.correct / self.total if self.total else float("nan")
def maybe_promote(champion: ArenaStats, challenger: ArenaStats, min_rounds=200, alpha=0.05):
if challenger.total < min_rounds or champion.total < min_rounds:
return False, "not enough rounds yet"
p_pool = (champion.correct + challenger.correct) / (champion.total + challenger.total)
se = np.sqrt(p_pool * (1 - p_pool) * (1 / champion.total + 1 / challenger.total))
z = (challenger.accuracy - champion.accuracy) / se
p_value = 1 - stats.norm.cdf(z) # one-sided: is challenger BETTER?
if p_value < alpha:
return True, f"challenger significantly better (z={z:.2f}, p={p_value:.4f})"
return False, f"not yet significant (z={z:.2f}, p={p_value:.4f})"A real run: a genuinely better challenger, tracked over time
A realistic setup: the champion was trained long ago on a small amount of early data (15 rows); the challenger benefits from a full, larger training set (400 rows) collected since. Both score the same live, labelled requests as they arrive.
from arena import *
X_train_small, y_train_small = make_stream(15, seed=0)
X_train_big, y_train_big = make_stream(400, seed=0)
CHAMPION = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train_small, y_train_small)
CHALLENGER = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train_big, y_train_big)
champ_stats, chal_stats = ArenaStats("champion"), ArenaStats("challenger")
X_live, y_live = make_stream(3000, seed=42)
for i in range(len(X_live)):
row, actual = X_live.iloc[[i]], int(y_live.iloc[i])
champ_stats.record(int(CHAMPION.predict(row)[0]), actual)
chal_stats.record(int(CHALLENGER.predict(row)[0]), actual)
if (i + 1) in (50, 200, 500, 1000, 2000, 3000):
promote, reason = maybe_promote(champ_stats, chal_stats)
print(f"after {i+1:>4} rounds: champion acc={champ_stats.accuracy:.4f} "
f"challenger acc={chal_stats.accuracy:.4f} promote={promote} ({reason})")after 50 rounds: champion acc=0.7200 challenger acc=0.9200 promote=False (not enough rounds yet) after 200 rounds: champion acc=0.7800 challenger acc=0.8650 promote=True (challenger significantly better (z=2.22, p=0.0131)) after 500 rounds: champion acc=0.8020 challenger acc=0.8660 promote=True (challenger significantly better (z=2.72, p=0.0033)) after 1000 rounds: champion acc=0.8090 challenger acc=0.8610 promote=True (challenger significantly better (z=3.13, p=0.0009)) after 2000 rounds: champion acc=0.8090 challenger acc=0.8595 promote=True (challenger significantly better (z=4.29, p=0.0000)) after 3000 rounds: champion acc=0.8113 challenger acc=0.8593 promote=True (challenger significantly better (z=5.01, p=0.0000))
Line-by-line: the fifty-round result is the most important line
At 50 rounds, the challenger already looks dramatically better — 92% against 72%, a twenty-point gap. maybe_promote still refuses, because min_rounds has not been reached. That refusal is deliberate and correct: fifty requests is not enough evidence to trust a twenty-point gap, no matter how large it looks. By 200 rounds, the same real advantage is confirmed statistically (p=0.0131), and it stays confirmed, with growing confidence, all the way to 3000 rounds. The gap did not change much after round 200 — what changed was how sure the test could be that the gap was real.
Testing the promotion rule against three honest scenarios
from arena import ArenaStats, maybe_promote
def test_promotion_is_refused_before_the_minimum_round_count():
champion, challenger = ArenaStats("c"), ArenaStats("x")
for _ in range(50):
champion.record(1, 1)
challenger.record(1, 1)
for _ in range(50):
champion.record(0, 1) # champion starts missing
challenger.record(1, 1) # challenger keeps a perfect streak
promote, reason = maybe_promote(champion, challenger, min_rounds=200)
assert promote is False
assert "not enough rounds" in reason
def test_a_sustained_real_advantage_gets_promoted():
champion, challenger = ArenaStats("c"), ArenaStats("x")
for i in range(1000):
champion.record(1, 1 if i % 5 else 0) # champion wrong 1 in 5 -- 80% accurate
challenger.record(1, 1) # challenger always right -- 100% accurate
promote, _ = maybe_promote(champion, challenger, min_rounds=200)
assert promote is True
def test_a_worse_challenger_is_never_promoted_however_long_it_runs():
champion, challenger = ArenaStats("c"), ArenaStats("x")
for i in range(5000):
champion.record(1, 1) # champion always right
challenger.record(1, 1 if i % 3 else 0) # challenger wrong 1 in 3
promote, _ = maybe_promote(champion, challenger, min_rounds=200)
assert promote is Falsepytest test_arena.py -q... [100%] 3 passed in 0.91s
test_a_worse_challenger_is_never_promoted_however_long_it_runs deliberately runs for five thousand rounds. However long a genuinely worse challenger is watched, more evidence only makes the test more confident it is worse, never flips the decision toward promoting it.
Common mistakes
Promoting on the first sign of a lead. The fifty-round result above is the whole lesson in one line: an apparent advantage that has not cleared the minimum sample size is not evidence yet, however dramatic it looks.
Running only one challenger at a time. Champion/challenger scales naturally to several challengers scored against the same champion simultaneously — different architectures, different feature sets, different hyperparameters — which is often more informative than testing ideas one at a time, sequentially, over months.
Never retiring a losing challenger. A challenger that has run for a long time without reaching significance, and is not trending toward it, is answering its own question: it is probably not better. Retire it and free the resources it was using, rather than letting it run forever "in case it's needed someday".
Forgetting the promoted challenger needs its own rollout plan. Winning the arena earns a challenger the right to become the champion — it does not, on its own, replace how that switch actually happens. Promotion still goes through the same canary or blue-green mechanics as any other release.
Comparing on a metric nobody actually cares about. Accuracy is simple to demonstrate here; a real arena should score challengers on the metric the business actually depends on, plus the same guardrail metrics a canary would watch, so a challenger that wins on accuracy but quietly hurts something else is not promoted by mistake.
Try it yourself
Add a second challenger to the arena, trained with yet another configuration, and run all three simultaneously against the same live stream. Decide how maybe_promote should behave when two challengers both clear significance at the same round — promoting the wrong one of two genuine improvements is a real decision this extension forces you to make on purpose.
What to learn next
- Swapping weights with no downtime — the mechanics of actually installing a newly promoted challenger as champion.
- Versioning a model API — keeping the promoted model's interface stable for whatever called the previous champion.
- Peeking and sequential testing — the statistically rigorous version of the repeated-checking pattern this arena uses in simplified form.
Researcher — Mathematics and papers.
Champion/challenger as sequential hypothesis testing
Checking maybe_promote at several points as data accumulates (50, 200, 500, ... rounds above) is, exactly as with a ramping canary, a form of repeated testing against a fixed significance threshold, and inherits the same inflated false-positive risk discussed in peeking and sequential testing. A rigorous implementation uses a sequential testing procedure with pre-specified stopping boundaries (a group-sequential design, or an always-valid p-value construction) rather than checking a fixed-sample z-test at arbitrary intervals, precisely because the arena above checks at several points and a naive fixed-$\alpha$ test evaluated repeatedly does not control the overall false-positive rate at the nominal level.
Multi-armed bandit framings
With several challengers running simultaneously, the promotion decision is a special case of the best-arm identification problem in multi-armed bandits: given a fixed exploration budget (or a required confidence level), identify the arm (model) with the highest true accuracy, while controlling the probability of selecting a suboptimal one. Thompson sampling and UCB-based methods can additionally shift traffic toward promising challengers during the evaluation window itself, trading a purely observational arena (every model sees the same distribution of requests, as implemented here) for one that also reduces the cost of evaluating a decisively-losing challenger.
Delayed and partial outcome labels
The developer example above assumes ground truth (y_live) is known immediately, which is realistic for some domains (a same-session click) and unrealistic for others (a loan default, observed months later). Delayed labels covers the general problem; for champion/challenger specifically, the practical consequence is that ArenaStats.total for both models grows only as fast as labels actually arrive, which can make the "sustained" window in a promotion decision considerably longer in wall-clock time than the round counts shown here suggest.
Interaction with concept drift
A challenger judged against a champion over a long window is implicitly assuming the comparison itself remains fair throughout — that neither model's relative advantage is an artefact of a particular period's data distribution. If the underlying data drifts partway through a long arena run, an early advantage can erode or reverse for reasons unrelated to which model is genuinely better; segmenting the promotion analysis by time window, and checking for concept drift directly, guards against attributing a distribution shift to a model quality difference.
Papers
- Lai and Robbins, Asymptotically Efficient Adaptive Allocation Rules, Advances in Applied Mathematics 1985 — the foundational multi-armed bandit result underlying best-arm identification framings of this problem.
- Johari et al., Always Valid Inference: Continuous Monitoring of A/B Tests, Operations Research 2022 — the sequential-testing correction this lesson's simplified fixed-$\alpha$ check should be upgraded to in a real, continuously-checked arena.
What to learn next
- Swapping weights with no downtime — the mechanics of actually installing a newly promoted challenger as champion.
- Versioning a model API — keeping the promoted model's interface stable for whatever called the previous champion.
- Peeking and sequential testing — the statistically rigorous version of the repeated-checking pattern this arena uses in simplified form.