Learn AI Create the Future — with Pranay
Main site

Caching and Cost Control

The one-box production stack

For most teams, one well-built machine running the model, a cache and a request log together is simpler, cheaper and more reliable than a sprawl of separate services.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

A one-box stack runs the model, its cache and its request logging together on one machine. Nothing is spread across separate services.

The analogy you have already lived

A small dhaba runs its whole kitchen from one room. The stove, the fridge, the prep counter — all within arm's reach of one cook. Nothing has to travel between buildings. Nothing has to wait for a delivery from across town.

A huge hotel kitchen, by contrast, has a separate bakery, a separate cold store, a separate butchery. Each is run by a different team, connected by trolleys moving between them. That complexity earns its keep at huge scale. For a dhaba serving a hundred plates a day, it would be pure overhead — more coordination than cooking.

Why it exists

Every earlier lesson in this section added one useful piece — a cache, an invalidation rule, a cost calculation. Real systems are tempted to put each piece in its own separate service. One microservice for caching, one for the model, one for logging — all talking to each other over the network.

That split earns its cost at real scale, where each piece needs to grow independently. Below that scale, it is mostly extra network calls, extra failure points, extra things that can each go down separately. It solves a scaling problem you do not have yet.

How it works

   ONE machine, ONE process

   +----------------------------------------------+
   |   incoming request                            |
   |        |                                      |
   |        v                                      |
   |   check the cache  (in memory, no network hop) |
   |        |                                      |
   |    hit |    miss                               |
   |    |    v                                      |
   |    |   run the model  (right here, no hop)     |
   |    |    |                                      |
   |     \   v                                      |
   |      answer, AND log the request  (same box)   |
   +----------------------------------------------+

Every step happens inside one process. There is no separate cache server to lose contact with, no separate logging service that can silently fall behind.

A real example you have seen

Plenty of real, profitable products run this way for a long time, before ever needing anything more elaborate. A single well-specified server does the model, the cache and the logging itself. It is not a beginner's shortcut. It is often the correct, deliberate choice, right up until real evidence says otherwise.

The honest part

This stops being right at some point. One machine has a ceiling on how much traffic it can handle. A single machine going down takes everything with it. The skill is not "always use one box" or "always use many boxes." It is recognising which side of that line you are actually on. Use real evidence, not a guess about future scale you do not have yet.

Remember this

  • A one-box stack keeps the model, cache and logging together, on one machine.
  • It removes network hops and separate failure points that a split architecture adds.
  • It is the right choice below a real scale ceiling, and the wrong choice above it. Know which side you are on.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install fastapi "uvicorn[standard]" cachetools httpx

A working one-box service

one_box.py
import sqlite3
import time
from contextlib import asynccontextmanager

from cachetools import LRUCache
from fastapi import FastAPI
from pydantic import BaseModel

STATE = {}
COST_PER_MODEL_CALL = 0.01  # illustrative example rate, not a live price

def slow_model(x: float) -> float:
    time.sleep(0.02)  # pretend this is real, paid model work
    return round(x * x, 4)

@asynccontextmanager
async def lifespan(app: FastAPI):
    STATE["cache"] = LRUCache(maxsize=1000)
    # check_same_thread=False: FastAPI runs sync endpoints on worker threads,
    # not the thread that created this connection. See "common mistakes" below.
    STATE["db"] = sqlite3.connect(":memory:", check_same_thread=False)
    STATE["db"].execute("CREATE TABLE requests (x REAL, result REAL, cache_hit INTEGER, ts REAL)")
    yield
    STATE["db"].close()

app = FastAPI(lifespan=lifespan)

class Query(BaseModel):
    x: float

@app.get("/healthz")
def healthz():
    return {"status": "ok"}

@app.post("/predict")
def predict(q: Query):
    cache = STATE["cache"]
    hit = q.x in cache
    result = cache[q.x] if hit else slow_model(q.x)
    if not hit:
        cache[q.x] = result
    STATE["db"].execute("INSERT INTO requests VALUES (?, ?, ?, ?)", (q.x, result, int(hit), time.time()))
    STATE["db"].commit()
    return {"result": result, "cache_hit": hit}

@app.get("/stats")
def stats():
    db = STATE["db"]
    total = db.execute("SELECT COUNT(*) FROM requests").fetchone()[0]
    hits = db.execute("SELECT COUNT(*) FROM requests WHERE cache_hit=1").fetchone()[0]
    return {"total_requests": total, "cache_hits": hits, "estimated_cost_saved": round(hits * COST_PER_MODEL_CALL, 4)}

Exercise it without starting a server

check.py
from fastapi.testclient import TestClient
from one_box import app

with TestClient(app) as client:
    for x in [3.0, 5.0, 3.0, 3.0, 7.0, 5.0]:
        print(client.post("/predict", json={"x": x}).json())
    print("health:", client.get("/healthz").json())
    print("stats: ", client.get("/stats").json())
Output
{'result': 9.0, 'cache_hit': False}
{'result': 25.0, 'cache_hit': False}
{'result': 9.0, 'cache_hit': True}
{'result': 9.0, 'cache_hit': True}
{'result': 49.0, 'cache_hit': False}
{'result': 25.0, 'cache_hit': True}
health: {'status': 'ok'}
stats:  {'total_requests': 6, 'cache_hits': 3, 'estimated_cost_saved': 0.03}

Every line is a real run of real code. One process served predictions, cached repeats, logged every request to SQLite, and reported a cost estimate. No separate cache server, no separate logging service, anywhere in sight.

Line-by-line walkthrough

check_same_thread=False. Without this, the demo crashes with a real error the first time a request runs: SQLite objects created in a thread can only be used in that same thread. FastAPI runs synchronous endpoint functions on worker threads, not the thread lifespan ran on. This flag is what makes the shared connection safe to use from any of them.

One STATE dictionary holds everything. The cache and the database connection live side by side. Both are loaded once at startup, exactly like the model loading pattern from model serving. Nothing here talks over a network to reach either one.

/stats reads directly from the same SQLite file the requests were logged into. No separate analytics pipeline. No delay before the numbers are available — the request that just happened is visible in the very next call.

Common mistakes

Reaching for Redis and a message queue before you have evidence you need them. Every extra service is one more thing that can be down while your model is up. It is also one more network hop on every request. Add a separate service when a measurement says this box cannot keep up — not before.

Using SQLite as if it were built for heavy concurrent writes. It handles the read-heavy, moderate-write pattern here comfortably. A system logging thousands of writes per second from many processes needs a database actually built for that. Not a bigger hammer applied to the wrong nail.

No plan for what happens when the one box goes down. Simplicity is no excuse to skip a health check, a restart policy, or a backup of what SQLite has logged. Health checks still matter here, exactly as much as in a larger architecture.

Treating "one box" as permanent instead of as a stage. The point is not that this scales forever. The point is that it is the right amount of complexity for where most systems actually are. Growing past it should be a decision made with evidence, not a default started with on day one.

Try it yourself

Point sqlite3.connect(":memory:", ...) at a real file path instead, and restart the process. Confirm the request log from before the restart is still there. That persistence is most of what a "real" logging service buys you, for free, in one line.

What to learn next

Researcher — Mathematics and papers.

Complexity has a real, measurable cost

Every additional service in an architecture adds a network call with its own latency distribution. It adds its own failure mode too (see the tail at scale for why tail latency compounds across calls), and its own operational burden — monitoring, deployment, on-call ownership. A system of $n$ independently-failing services with per-service availability $a$ has combined availability $a^n$ if a request depends on all of them — meaning added services must each contribute enough value to be worth the multiplicative reliability cost, not be added for fashion alone.

This is the reasoning behind treating "one box" not as an unsophisticated default, but as a considered choice with its own name in the literature: a monolith. In the modern sense, that means a well-structured single deployable unit — distinct from a poorly-structured single codebase. Fowler and Lewis's foundational writing on microservices (2014) explicitly recommends starting with a monolith. Extract services only along boundaries that real operational pain has already revealed — "monolith first," not microservices by default.

The actual scaling ceiling of a single box

A single well-specified machine is not as limited as intuition suggests. A modern multi-core server with an in-process cache and SQLite in WAL mode can sustain several thousand requests per second for lightweight prediction workloads. That is well beyond what most products, even successful ones, ever see in practice. The ceiling that actually matters is usually not raw throughput. It is one of: model memory footprint exceeding one machine, a genuine need for zero-downtime deploys of very large state, or a hard requirement for geographic distribution close to users worldwide. None of those are default conditions; each should be demonstrated, not assumed.

When to split, and along what boundary

When splitting genuinely becomes necessary, the correct axis is usually independent scaling need, not "separate concerns" in the abstract. A cache that needs to scale independently of the model (shared across many model instances, for instance) is a legitimate reason to extract it to Redis. A logging pipeline that needs to survive the model process crashing is a legitimate reason to ship logs elsewhere. "It felt cleaner to split them" is not, on its own, a reason grounded in an actual constraint.

Reading

  • Fowler and Lewis, Microservices, martinfowler.com, 2014 — the "monolith first" argument in its original form
  • Newman, Building Microservices (2nd ed.), O'Reilly, 2021 — chapter 1 covers the decomposition trade-offs directly

What to learn next