Learn AI Create the Future — with Pranay
Main site

Scaling and Traffic Management

Circuit breakers and fallbacks

A circuit breaker stops calling a dependency that is failing, and switches to a backup answer, instead of making every user wait for the same timeout.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A circuit breaker stops calling a service that is failing, and switches to a backup answer. No user has to wait through the same timeout.

The analogy you have already lived

You have seen a fuse trip in a home electrical box. The fuse does not wait for wires to overheat and catch fire. It cuts the circuit the moment current gets dangerous, protecting everything downstream from a fault upstream.

You have also kept a candle or a torch ready for a power cut. When the power genuinely goes, you do not sit in the dark waiting for it to come back. You reach for the backup, right away.

Why it exists

Every model server depends on other things: a database, another service, sometimes another model. One of those dependencies can start failing — slow, erroring, or both. The naive response is to keep calling it exactly as before. Every user then waits out the same slow failure, one at a time.

That naive response makes a bad situation worse. Every waiting request is also holding a thread, a connection, a slice of memory. Those resources pile up while nothing useful happens, sometimes badly enough to take down the caller too.

How it works

A circuit breaker has three states, named after the electrical fuse it borrows its name from.

   CLOSED (normal)
      |  calls flow through
      |  failures build up...
      v
   OPEN (tripped)
      |  calls skip the dependency entirely, go straight to a fallback
      |  after a cooldown period...
      v
   HALF-OPEN (testing)
      |  one real call is let through, to check if the dependency recovered
      |
      +-- succeeded --> back to CLOSED
      +-- failed -----> back to OPEN, cooldown restarts

A fallback is the backup answer used while the breaker is open. It might be a cached previous result, a safe default, or a clear "temporarily unavailable" instead of a hang.

A real example you have seen

A shopping app that shows "reviews are temporarily unavailable" instead of a spinning wheel that never finishes is using a fallback. The main purchase flow keeps working. One broken dependency did not take the whole page down with it.

The honest part

A circuit breaker does not fix the failing dependency. It contains the damage. That buys the dependency time to recover, and protects everyone else from feeling every second of it. Whatever caused the outage still needs fixing on its own.

Remember this

  • A circuit breaker stops calling a failing dependency, instead of letting every caller wait out the same timeout.
  • Open means calls are short-circuited to a fallback; half-open is a careful test of whether recovery has happened.
  • A fallback — a cached answer, a safe default — keeps the rest of the system working while the breaker is open.

What to learn next

Developer — Code and libraries.

Setup

No installs needed beyond the standard library.

A working circuit breaker, against a real flaky dependency

circuit_breaker.py
import time


class CircuitBreaker:
    """closed: calls flow through. open: calls skip straight to the
    fallback. half_open: one call is let through to test recovery."""

    def __init__(self, fail_threshold=3, cooldown=0.12):
        self.fail_threshold = fail_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.state = "closed"
        self.opened_at = None

    def call(self, fn, fallback):
        if self.state == "open":
            if time.perf_counter() - self.opened_at >= self.cooldown:
                self.state = "half_open"
            else:
                return fallback(), "short_circuited"
        try:
            result = fn()
        except RuntimeError:
            self.failures += 1
            if self.state == "half_open" or self.failures >= self.fail_threshold:
                self.state, self.opened_at = "open", time.perf_counter()
            return fallback(), "failed"
        else:
            self.failures, self.state = 0, "closed"
            return result, "ok"


call_count = 0


def flaky_downstream():
    """Down (slow, then errors) for the first 5 calls; healthy after that."""
    global call_count
    call_count += 1
    time.sleep(0.05)  # stands in for a real timeout, shortened to keep this fast
    if call_count <= 5:
        raise RuntimeError("downstream unavailable")
    return "live result"


def fallback():
    return "cached fallback value"  # instant: no network call at all


def run(use_breaker):
    global call_count
    call_count = 0
    breaker = CircuitBreaker() if use_breaker else None
    outcomes = {"ok": 0, "failed": 0, "short_circuited": 0}
    t0 = time.perf_counter()
    for _ in range(30):
        if use_breaker:
            _, status = breaker.call(flaky_downstream, fallback)
        else:
            try:
                flaky_downstream()
                status = "ok"
            except RuntimeError:
                status = "failed"
        outcomes[status] += 1
        time.sleep(0.02)  # spacing between arriving requests
    elapsed = time.perf_counter() - t0
    return elapsed, outcomes, call_count


for label, use_breaker in [("no breaker", False), ("with breaker", True)]:
    elapsed, outcomes, downstream_calls = run(use_breaker)
    print(f"{label:13s} total time: {elapsed:5.2f}s   downstream calls made: {downstream_calls:2d}"
          f"   outcomes: {outcomes}")
Output
no breaker    total time:  2.13s   downstream calls made: 30   outcomes: {'ok': 25, 'failed': 5, 'short_circuited': 0}
with breaker  total time:  1.38s   downstream calls made: 15   outcomes: {'ok': 10, 'failed': 5, 'short_circuited': 15}

Real wall-clock timing is involved (the time.sleep calls), so total-time numbers will vary slightly by machine load. The pattern is stable across runs on the same machine, though — the breaker version takes noticeably less time, and makes about half the real downstream calls.

Walking through it

The unprotected version pays the full 0.05-second cost of every single call, even the five guaranteed to fail. 30 calls, 30 real waits, no shortcuts.

The breaker version makes only 15 real downstream calls, out of 30 total requests. After 3 consecutive failures, it opens and short-circuits to the fallback — instantly, no wait — until the cooldown passes and it cautiously tries again.

It recovers on its own. Once call_count passes 5 during one of those half-open probes, flaky_downstream starts succeeding. The breaker closes back to normal, visible in the 'ok': 10 outcomes — all happening after the dependency actually healed.

5 failures still reach the caller in both versions. The breaker does not make the underlying outage vanish. It stops repeating the same failure once the pattern is clear, replacing 15 more slow failures with 15 instant fallbacks.

Common mistakes

Catching every exception, not only the ones that mean "unhealthy." A validation error is the caller's fault, not the dependency's. Counting it as a failure can trip the breaker over requests that were never going to succeed anyway.

No fallback worth having. A circuit breaker that opens and returns nothing useful only converts a slow failure into a fast one. A fallback that actually helps the caller — a cached value, a sensible default — is what makes the pattern worth the complexity.

One shared breaker for many unrelated dependencies. A single breaker guarding both a database call and an unrelated API call trips on either one. It then short-circuits calls that had nothing to do with the actual failure. Scope one breaker per dependency.

Never revisiting the threshold and cooldown. Values copied from an example — including this one — are a starting point, not a fact about your system. Tune them against your dependency's real failure and recovery patterns.

Try it yourself

Change fail_threshold to 1 and rerun. Watch the breaker open after a single failure instead of three. That trades a faster reaction for a much higher chance of tripping on an isolated blip that would have recovered on its own.

What to learn next

Researcher — Mathematics and papers.

Half-open concurrency

The implementation above lets exactly one call through per half-open period, sequentially, because the demo itself is sequential. A concurrent system faces a real design question. If ten requests arrive during the half-open window, does exactly one probe the dependency while the other nine wait or short-circuit? Or do all ten probe at once, risking exactly the pile-up the breaker exists to prevent? Netflix's Hystrix answers this with a semaphore limiting concurrent half-open probes to a small fixed number, not necessarily one. Hystrix itself is now in maintenance, but remains architecturally influential — resilience4j and Polly are its direct successors in the JVM and .NET ecosystems respectively.

Distinguishing failure types

A production breaker typically distinguishes fast failures (connection refused, immediate error) from slow failures (timeout after waiting the full budget). Counting only fast failures toward the trip threshold, while separately tracking timeout rate, avoids a subtle trap. Otherwise the breaker takes as long to trip as the dependency itself takes to time out, defeating the purpose of tripping early.

The bulkhead pattern, and how it composes with circuit breakers

A bulkhead — named for a ship's watertight compartments — limits how many concurrent calls to one dependency are allowed at all, regardless of failure state. One slow dependency then cannot exhaust the caller's entire thread or connection pool, even before enough failures accumulate to trip a breaker. Circuit breakers and bulkheads are complementary, not alternatives: a bulkhead bounds concurrent exposure; a breaker reacts to observed failure. Nygard's Release It! (2007, 2nd ed. 2018) is the primary reference that named and popularised both patterns together as a coherent resilience toolkit for distributed systems.

Formal treatment as a Markov process

The three-state breaker is a small Markov chain, with transition probabilities driven by the dependency's underlying failure rate $p$. For a fixed threshold $k$ and cooldown $\tau$, the expected time spent in the open state before a successful half-open probe grows with $p$ and shrinks with $\tau$. This formalises the intuitive trade-off: recovering quickly once healthy, against risking a hammered dependency that has not actually healed.

References

  • Nygard, Release It!: Design and Deploy Production-Ready Software, 2nd edition, Pragmatic Bookshelf, 2018 — the standard reference for circuit breakers, bulkheads and the broader stability-pattern vocabulary.
  • Netflix Technology Blog, Introducing Hystrix for Resilience Engineering, 2012 — the industry implementation that made the pattern mainstream outside its original Erlang/telecom roots.

What to learn next