Learn AI Create the Future — with Pranay
Main site

Serving Models in Production

Serving a model with FastAPI

Taking a FastAPI model server from working to production-ready means understanding how it handles many requests arriving at once, not repeating the basics of loading and validation.

Read these first

On this page 7
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A production FastAPI server has to answer many requests arriving at the same moment, not only one at a time.

The analogy you have already lived

Picture a restaurant with one waiter. Say that waiter stands at the kitchen window until each dish is ready before taking the next order. Tables then wait far longer than they should. A waiter who takes the next order while the kitchen works serves many tables at once, with no extra staff.

A model server is that waiter. This lesson is about which style of waiter you have written, often without realising you chose.

Why it exists

Model serving already covered the essentials: load the model once, validate every request, add a health check. Those apply here unchanged — read that lesson first if you have not.

What it did not cover: real traffic does not arrive one request at a time. Tens or hundreds of requests can land in the same second. How your server behaves in that moment is a separate question from whether it answers one request correctly.

How it works

ONE waiter who blocks at the kitchen window:
  request 1 arrives -> waits for the "kitchen" -> answered
                        (request 2 cannot even start yet)
  request 2 arrives -> waits -> answered
  total time for 2 requests: roughly DOUBLE one request's time

ONE waiter who hands off to the kitchen and keeps moving:
  request 1 arrives -> handed to a worker -> waiter free again
  request 2 arrives -> handed to a worker -> waiter free again
  total time for 2 requests: roughly the SAME as one request

FastAPI can do either, depending on one keyword in your code: async def versus plain def. Getting this wrong is invisible in a quick local test and expensive under real load.

A real example you have seen

A UPI app during a payday rush, when thousands of people check their balance within the same minute. An app that handles one person at a time, blocking on each, would feel frozen for everyone else waiting behind them.

Remember this

  • A model server's job is answering many requests overlapping in time, not only one at a time correctly.
  • FastAPI can run your code two different ways — one lets requests overlap, one does not.
  • The difference is invisible with one test request, and very visible under real traffic.

What to learn next

  • Model serving — the fundamentals this lesson assumes: loading once, validating input, health checks.
  • Warm-up and cold starts — what happens in the seconds before a server is ready to handle any of this.
  • gRPC vs REST for inference — a different axis of the same "how fast can this server really go" question.

Developer — Code and libraries.

Setup

bash
pip install fastapi "uvicorn[standard]" httpx

The mistake, made visible

This uses FastAPI's own in-process test transport to call ten "requests" at once against two endpoints that do the identical blocking work, differing only in how that work is declared.

concurrency_demo.py
import asyncio
import time

import httpx
from fastapi import FastAPI

app = FastAPI()


def slow_feature_lookup():
    # Stands in for a blocking call: a database read, a feature-store hit.
    time.sleep(0.2)
    return "ok"


@app.get("/predict-bad")
async def predict_bad():
    # async def, but the body blocks the event loop directly. The mistake.
    slow_feature_lookup()
    return {"status": "ok"}


@app.get("/predict-good")
def predict_good():
    # plain def: FastAPI runs this on a worker thread automatically.
    slow_feature_lookup()
    return {"status": "ok"}


async def fire(path, n):
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
        start = time.perf_counter()
        await asyncio.gather(*[client.get(path) for _ in range(n)])
        return time.perf_counter() - start


async def main():
    n = 10
    bad = await fire("/predict-bad", n)
    good = await fire("/predict-good", n)
    print(f"n_requests={n}")
    print(f"async def, blocking body  (/predict-bad) : {bad:.2f}s total")
    print(f"plain def, auto threadpool (/predict-good): {good:.2f}s total")

asyncio.run(main())
Output
n_requests=10
async def, blocking body  (/predict-bad) : 2.01s total
plain def, auto threadpool (/predict-good): 0.21s total

Both endpoints call the exact same slow_feature_lookup, which always takes 0.2 seconds. Ten calls one after another would take 2 seconds — which is exactly what /predict-bad gives you. /predict-good gives you back roughly one call's worth of time, because the ten calls overlapped.

Line-by-line walkthrough

/predict-bad is declared async def, which tells FastAPI "trust me, this will not block the event loop". Then its body calls time.sleep(), which blocks it directly and completely — every other request waits behind this one, on the same single thread.

/predict-good is declared as a plain def. FastAPI's rule for plain def path functions is to automatically run them on a worker thread from a pool, specifically so a blocking call inside them does not freeze anyone else. Nothing else in the code changed — only the keyword.

Common mistakes

Writing async def out of habit, everywhere, without an await that actually yields. async def is a promise about behaviour, not a performance switch. If the body never awaits anything, it blocks exactly like a synchronous function, with extra confusion added on top.

Calling a genuinely slow synchronous library (a blocking database driver, requests) inside an async def handler. This is the exact bug above, with a real library instead of time.sleep. Either use that library's async version, or declare the handler as plain def so FastAPI threads it for you.

Assuming --workers 4 fixes this. More worker processes help with CPU-bound load, covered in warm-up and cold starts, but they do not fix one process blocking its own event loop — that is a code-shape problem, not a scaling-number problem.

Try it yourself

Change time.sleep(0.2) to a genuinely async-friendly await asyncio.sleep(0.2) inside /predict-bad, keeping it async def. Re-run, and confirm it now performs the same as /predict-good — because now it is actually yielding control, instead of only claiming to.

What to learn next

  • Model serving — the fundamentals this lesson assumes: loading once, validating input, health checks.
  • Warm-up and cold starts — what happens in the seconds before a server is ready to handle any of this.
  • gRPC vs REST for inference — a different axis of the same "how fast can this server really go" question.

Researcher — Mathematics and papers.

The ASGI concurrency model

FastAPI runs on an ASGI server (Uvicorn), backed by a single-threaded event loop per worker process. An async def path function runs directly on that event loop. A plain def path function is automatically dispatched to a bounded thread pool via Starlette's run_in_threadpool, so it cannot block the loop that every other in-flight request also depends on.

This means the two endpoint styles are not interchangeable performance-wise: async def is strictly better for I/O-bound work that uses a genuinely async library (an async database driver, httpx instead of requests), since it avoids thread-pool scheduling overhead entirely. Plain def is the safer default for CPU-bound work or for wrapping a library with no async support, since the thread pool bounds how much damage one slow call can do.

Process-level scaling

A single Uvicorn worker is limited by the GIL for CPU-bound Python code — the global interpreter lock, which permits only one thread to run Python bytecode at a time even across a thread pool. uvicorn --workers N starts N separate OS processes, each with its own event loop, GIL and full copy of the loaded model in memory. This is the correct lever for CPU-bound throughput; the async/threadpool distinction above is the correct lever for I/O-bound concurrency within one process.

Backpressure and admission control

Neither concurrency style above bounds how many requests can be in flight simultaneously. Under a sudden spike, both an unbounded thread pool queue and an unbounded event-loop task queue will accept more work than the machine can complete promptly, growing memory and worsening tail latency for everyone already waiting — the queueing behaviour described formally in model serving's researcher section via Little's law.

Production servers add explicit admission control: a semaphore or bounded queue in front of the handler, returning 503 once a concurrency ceiling is reached rather than accepting unboundedly and degrading everyone's latency together. This trades a small number of explicit, fast failures for protecting the latency of everything else.

Papers and systems

  • Dean and Barroso, The Tail at Scale, CACM 2013 — already central to model serving; the concurrency choices here are what determine whether a service's queueing behaviour matches that paper's model.
  • The ASGI specification itself — asgi.readthedocs.io — is the authoritative source for exactly what guarantees async def versus threaded def provide.
  • Fowler, CircuitBreaker, and the broader bulkhead-pattern literature — the design basis for admission control and concurrency ceilings described above.

What to learn next

  • Model serving — the fundamentals this lesson assumes: loading once, validating input, health checks.
  • Warm-up and cold starts — what happens in the seconds before a server is ready to handle any of this.
  • gRPC vs REST for inference — a different axis of the same "how fast can this server really go" question.

Related terms

What to learn next

These follow on from what you just read.

  • Serving Models in Production

    NVIDIA Triton inference server

    Triton is a dedicated inference server that handles dynamic batching, multiple frameworks and GPU scheduling for you, instead of you hand-rolling that logic inside a web framework.

  • Serving Models in Production

    BentoML

    BentoML is a Python framework that turns a trained model into a packaged, servable API with far less boilerplate than writing every route and Dockerfile by hand.

  • Serving Models in Production

    Ray Serve

    Ray Serve runs several copies of a model across many processes or machines behind one address, scaling a service by adding replicas instead of rewriting it.