Learn AI Create the Future — with Pranay
Main site

Incident Response for ML Systems

Surviving a provider outage

When a service you depend on goes down, the goal is not to fix it — you cannot — it is to keep your own system standing while it recovers.

Read these first

On this page 9
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. Retrying is not free
  6. Where you have already felt this
  7. The honest part
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A provider outage is a service you depend on going down — your job is keeping your own system standing.

The analogy you have already lived

You have lived through a power cut. The lights go out. If your home has an inverter, it kicks in a second later — dimmer, but working. You do not call the power company and demand they hurry. You switch to the backup, and wait.

A provider is an outside service your system depends on — an LLM API, a payments gateway, a maps service. When it goes down, your job is the inverter, not the repair crew.

Why it exists

Modern ML systems rarely stand alone. A chatbot calls a hosted language model. A fraud check calls an address-verification API. A recommendation service calls a feature store owned by another team.

Every one of those is a promise made by somebody else's infrastructure. Every promise eventually breaks — a data centre issue, a rate limit, a bad deploy on their side. When it does, the question is not "how do I stop this." It is "how does my system behave for the next twenty minutes."

How it works

Normal:      you  --request-->  provider  --answer-->  you

Outage:      you  --request-->  provider  --timeout / error

Bad response to that:
      you keep calling, again and again, as fast as before
      -> the provider, already struggling, gets hit harder
      -> your own system fills up waiting on calls that will not return

Good response to that:
      you notice the pattern of failures
      -> stop calling for a while
      -> answer from a cache, a default, or "please try again shortly"
      -> check back occasionally to see if it has recovered

The bottom path has two ideas worth naming. A circuit breaker stops sending requests to something already known to be broken. A fallback is what you answer with instead, while it stays broken.

Retrying is not free

The first instinct is to retry a failed call. That is correct, in moderation — a single dropped packet is common and a second attempt often succeeds.

Retrying instantly and endlessly is not moderation. If a thousand requests retry at once, a struggling provider becomes a shut-down one, and your outage gets worse. Wait a little longer between each retry — this is called backoff — and eventually stop retrying and fall back.

Where you have already felt this

A ride-hailing app that still shows a price when the live-traffic service is down. It uses yesterday's estimate instead of refusing to work. A payment page that says "try again in a moment" instead of hanging forever.

The honest part

You cannot rehearse every real outage. A real provider going down is exactly the infrastructure you do not control. So this lesson teaches the pattern with a simulated failure instead. The pattern transfers directly — only the network calls are pretend.

Remember this

  • A provider outage is not yours to fix. Your job is keeping your own system usable while it lasts.
  • Stop hammering a struggling provider — retry a little, with backoff, then fall back instead.
  • A circuit breaker remembers recent failures and stops calling a provider that is already down.

What to learn next

Developer — Code and libraries.

Setup

None. Pure Python standard library.

A circuit breaker with a cached fallback

circuit_breaker.py
class CircuitBreaker:
    """CLOSED: calls go through. OPEN: calls fail fast, no attempt made.
    HALF_OPEN: one probe call is allowed through to test recovery.

    A real breaker uses a wall-clock cooldown ("wait 30 seconds before
    probing again"). This one counts calls instead of seconds, so the demo
    is deterministic and instant to run — the logic is identical either way."""

    def __init__(self, failure_threshold: int = 3, cooldown_calls: int = 3):
        self.failure_threshold = failure_threshold
        self.cooldown_calls = cooldown_calls
        self.state = "CLOSED"
        self.consecutive_failures = 0
        self.calls_since_opened = 0

    def call(self, provider_fn, fallback_fn, *args):
        if self.state == "OPEN":
            self.calls_since_opened += 1
            if self.calls_since_opened >= self.cooldown_calls:
                self.state = "HALF_OPEN"
            else:
                return fallback_fn(*args), "OPEN (skipped the provider entirely)"

        try:
            result = provider_fn(*args)
        except RuntimeError:
            self.consecutive_failures += 1
            if self.state == "HALF_OPEN" or self.consecutive_failures >= self.failure_threshold:
                self.state = "OPEN"
                self.calls_since_opened = 0
            return fallback_fn(*args), f"{self.state} (provider call failed)"

        self.state = "CLOSED"
        self.consecutive_failures = 0
        return result, "CLOSED (provider answered)"


# A simulated provider: down for calls 3-9, back up after that.
# Fixed, not random, so this incident is reproducible for anyone who runs it.
DOWN_DURING = range(3, 10)


def flaky_provider(call_number: int, city: str) -> str:
    if call_number in DOWN_DURING:
        raise RuntimeError("provider timeout")
    return f"live-geocode({city})"


def cached_fallback(call_number: int, city: str) -> str:
    return f"cached-geocode({city})"


breaker = CircuitBreaker(failure_threshold=3, cooldown_calls=3)

for call_number in range(12):
    result, path = breaker.call(flaky_provider, cached_fallback, call_number, "Pune")
    print(f"call {call_number:>2}: {path:<38} -> {result}")
Output
call  0: CLOSED (provider answered)             -> live-geocode(Pune)
call  1: CLOSED (provider answered)             -> live-geocode(Pune)
call  2: CLOSED (provider answered)             -> live-geocode(Pune)
call  3: CLOSED (provider call failed)          -> cached-geocode(Pune)
call  4: CLOSED (provider call failed)          -> cached-geocode(Pune)
call  5: OPEN (provider call failed)            -> cached-geocode(Pune)
call  6: OPEN (skipped the provider entirely)   -> cached-geocode(Pune)
call  7: OPEN (skipped the provider entirely)   -> cached-geocode(Pune)
call  8: OPEN (provider call failed)            -> cached-geocode(Pune)
call  9: OPEN (skipped the provider entirely)   -> cached-geocode(Pune)
call 10: OPEN (skipped the provider entirely)   -> cached-geocode(Pune)
call 11: CLOSED (provider answered)             -> live-geocode(Pune)

This is one deterministic run of the code above — no network, no randomness, so it reproduces identically every time it runs.

Reading the trace

Calls 0-2 succeed normally. Calls 3-4 fail, but the breaker is still CLOSED — it needs failure_threshold=3 consecutive failures before it gives up, so it keeps trying.

Call 5 is the third consecutive failure. The breaker opens. Calls 6-7 do not even attempt the provider — that is the entire point, protecting a struggling service from more load.

Call 8 is the interesting one: after cooldown_calls=3 calls in the OPEN state, the breaker allows one probe through. The provider is still down (call 8 is inside DOWN_DURING), so it fails, and the breaker snaps straight back to OPEN for another cooldown.

Call 11's probe finally lands after the provider recovered (DOWN_DURING ends at 9), and the breaker closes.

Backing off between retries

Backoff is the piece that runs before a breaker gives up — a handful of increasingly patient retries.

backoff.py
def backoff_schedule(max_attempts: int = 5, base_seconds: float = 0.5, cap_seconds: float = 8.0):
    return [min(cap_seconds, base_seconds * (2 ** attempt)) for attempt in range(max_attempts)]


for i, delay in enumerate(backoff_schedule()):
    print(f"attempt {i + 1}: wait {delay:.1f}s before retrying")
Output
attempt 1: wait 0.5s before retrying
attempt 2: wait 1.0s before retrying
attempt 3: wait 2.0s before retrying
attempt 4: wait 4.0s before retrying
attempt 5: wait 8.0s before retrying

This prints a schedule, not a live retry loop — no time.sleep runs here, so there is nothing timing-dependent to measure. Each delay doubles the last, capped at 8 seconds, so a struggling provider gets breathing room that grows instead of a hundred instant retries arriving at once.

Common mistakes

Retrying forever with no cap. An unbounded retry loop against a dead provider ties up your own resources indefinitely. Always pair retries with a maximum attempt count and a fallback.

No fallback at all. A circuit breaker that opens and then has nothing to return is a more polite way of failing, not a fix. cached_fallback above is the part that keeps the caller usable.

Treating every error as retryable. A 400 Bad Request will fail identically on every retry — it is your request that is wrong, not a transient provider hiccup. Retry transient errors (timeouts, 5xx, connection resets); do not retry client errors.

Testing the breaker only with a provider that never recovers. A breaker that opens correctly but never verifies it can close again has only been half tested. The probe-then-close behaviour at call 11 above is worth testing on its own.

Try it yourself

Change DOWN_DURING to range(3, 20) — a provider that never comes back inside this run — and confirm the breaker keeps probing every cooldown_calls and keeps falling back, without ever getting stuck open forever or closed forever.

What to learn next

Researcher — Mathematics and papers.

The circuit breaker pattern, formally

The pattern above follows Fowler's widely-cited description (Nygard, Release It!, 2007, popularised the term for software) as a three-state finite automaton:

  • Closed — requests pass through; failures increment a counter.
  • Open — requests fail immediately without attempting the call, for a cooldown period.
  • Half-open — a limited number of trial requests are allowed; success transitions to Closed, failure transitions back to Open.

The state machine's purpose is to bound the cost of a dependency failure. Without it, a synchronous call to a dead dependency can hold a thread or connection open until its own timeout fires — and under load, enough held connections exhaust a resource pool, a failure mode known as cascading failure, where one dependency's outage takes down callers that had no fault of their own.

Retry storms and correlated failure

A naive retry policy — fixed delay, no jitter, applied uniformly across many clients — produces a retry storm: when the provider recovers, every client that was waiting the same fixed interval retries at the same instant, re-creating the load spike that may have caused the original outage. AWS's architecture guidance (Brooker, "Exponential Backoff And Jitter," AWS Builders' Library) recommends adding randomised jitter to each backoff delay specifically to decorrelate retries across independent clients — "full jitter" (a uniformly random delay between zero and the exponential cap) empirically outperforms both fixed delay and un-jittered exponential backoff under load, in their reported benchmarks.

Bulkheads and timeout budgets

A circuit breaker protects a dependency; a bulkhead (the term borrowed from ship design, where a bulkhead is a compartment wall) protects your own system from a slow dependency by isolating its resource usage — a separate thread pool or connection limit per dependency, so one slow provider cannot exhaust the pool shared by every other call your service makes. The two patterns are complementary: a bulkhead bounds the damage while the breaker is still deciding whether the dependency is actually down.

A related discipline is a strict timeout budget per request: if a user-facing request has a 500ms total budget and a provider call inside it has no timeout of its own, that one dependency can consume the entire budget alone. Every outbound call needs an explicit timeout, chosen as a fraction of the total budget, not left to the provider's default.

Graceful degradation as a design goal

Netflix's public engineering writing on resilience (Hystrix, and its documented design principles, 2012-2018) frames the fallback path not as an afterthought but as a first-class piece of the design: for every dependency, decide in advance what a degraded-but-useful answer looks like, before an outage forces the decision under pressure. A stale cached value, a simplified rule-based approximation of the model, or an honest "temporarily unavailable" are all valid choices — the failure mode to avoid is discovering, during a real outage, that no fallback was ever designed.

What to learn next