Learn AI Create the Future — with Pranay
Main site

Caching and Cost Control

Caching embeddings

Turning text into embeddings costs real compute. If the same text gets embedded more than once, save the result the first time and reuse it.

Read these first

On this page 8
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. The honest part
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Once you have turned a piece of text into an embedding, save that result. Do not compute it again for the same text.

The analogy you have already lived

A librarian who writes a short summary card for every new book, and files it. The next time anyone asks what that book is about, the librarian reads the card. They do not reread the whole book from page one.

The card took real effort to write, once. After that, reusing it is nearly free. An embedding is that summary card, written in numbers instead of words.

Why it exists

An embedding is a list of numbers a model produces for a piece of text, capturing what the text means. You met this idea in semantic caching, where embeddings were used to match similar questions.

Computing an embedding is not free — it runs a model over the text. Many systems embed the same text repeatedly without realising it. A document gets re-indexed after every small website update. A product description gets embedded again every time a search happens. A support article gets embedded fresh on every deploy, even though nobody edited it.

How it works

   text: "Refunds take 5-7 business days."
              |
              v
   have we embedded THIS EXACT text before?
        |                          |
       yes                         no
        |                          |
   reuse the saved             run the embedding
   numbers (fast)               model, SAVE the result
              \                  /
               v                v
          numbers ready to use (for search, for a cache, for anything else)

The key used to check "have we seen this before" is usually a hash — a short fingerprint computed from the text. Two long identical texts can then be compared by comparing two short fingerprints, instead of the full text.

A real example you have seen

A search box on a shopping website returns results instantly. This happens even though "understanding" your query and every product description involves embeddings. The product descriptions were embedded once, when they were added to the catalogue. Not fresh, on every search, by every shopper.

The honest part

This cache is only correct as long as nothing meaningful about the text has changed. If a document is edited and the embedding is not recomputed, search results quietly start pointing at outdated meaning. Deciding when to invalidate this cache is its own real problem — the next lessons deal with it directly.

Remember this

  • Computing an embedding costs real compute; save the result rather than recomputing it.
  • Cache by a fingerprint of the exact text, so any change is detected automatically.
  • A cached embedding is only correct until the text it represents changes.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install sentence-transformers numpy

A persistent embedding cache

embedding_cache.py
import hashlib
import sqlite3
import time
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

db = sqlite3.connect(":memory:")  # use a real file path to persist across runs
db.execute("CREATE TABLE embeddings (text_hash TEXT PRIMARY KEY, vector BLOB)")

def text_key(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def embed(text: str) -> np.ndarray:
    key = text_key(text)
    row = db.execute("SELECT vector FROM embeddings WHERE text_hash = ?", (key,)).fetchone()
    if row is not None:
        return np.frombuffer(row[0], dtype=np.float32)
    vector = model.encode([text])[0].astype(np.float32)
    db.execute("INSERT INTO embeddings VALUES (?, ?)", (key, vector.tobytes()))
    db.commit()
    return vector

documents = [
    "Refunds take 5-7 business days.",
    "Our support line is open 9am to 6pm.",
    "Refunds take 5-7 business days.",  # a repeat, e.g. re-indexed after a re-crawl
]
for doc in documents:
    start = time.perf_counter()
    vec = embed(doc)
    ms = (time.perf_counter() - start) * 1000
    print(f"{ms:6.1f} ms   shape={vec.shape}   {doc!r}")

print("rows stored:", db.execute("SELECT COUNT(*) FROM embeddings").fetchone()[0])
Output
 233.2 ms   shape=(384,)   'Refunds take 5-7 business days.'
  10.7 ms   shape=(384,)   'Our support line is open 9am to 6pm.'
   0.0 ms   shape=(384,)   'Refunds take 5-7 business days.'
rows stored: 2

The first call is slower on every run of this script. It includes the embedding model finishing its own warm-up, not only the text itself. Real timings from this machine; expect different absolute numbers on yours.

Line-by-line walkthrough

sqlite3.connect(":memory:"). Used here so the example is fully self-contained and leaves nothing on disk. Change this to a file path (sqlite3.connect("embeddings.db")) to make the cache survive between separate runs of your program. That is the entire point in real use.

hashlib.sha256(...).hexdigest(). Turns arbitrarily long text into a fixed-length fingerprint. Two different texts producing the same hash is astronomically unlikely at this hash length. That is what makes it safe to use as a cache key.

Storing the vector as BLOB. vector.tobytes() turns the NumPy array into raw bytes SQLite can store directly; np.frombuffer(..., dtype=np.float32) reverses it. Getting the dtype wrong here silently returns garbage numbers, not an error — always fix the dtype at both ends.

Common mistakes

Hashing the wrong thing. If you hash text.lower() but embed the original text, a cache hit can return the wrong embedding whenever case differs meaningfully. Hash exactly what you feed to the model.

Never invalidating on edits. A cache keyed only by, say, a database row ID, rather than the text's own content, silently serves the old embedding after that row is edited. Keying by a hash of the text itself, as above, fixes this automatically. A changed text produces a cache miss on its own.

Storing floats as text. Serialising a vector with str(vector.tolist()) works, but wastes space and CPU time on parsing compared to raw bytes. It gets worse the more vectors you store.

Assuming the model never changes. If you switch embedding models, every previously cached vector was produced by a different model. It is no longer comparable to new ones. Include the model's name and version in the cache key, or clear the cache on a model change.

Try it yourself

Add a model_version column to the table, include it in the key, and store "all-MiniLM-L6-v2" for every row. Then imagine — and write a comment describing — what breaks if you skip this and swap embedding models later.

What to learn next

Researcher — Mathematics and papers.

Cost accounting for embedding caches

Let $N$ be the number of embedding calls a workload makes without caching, and $r$ the fraction of those calls that are exact repeats of previously seen text. The number of embedding-model invocations with caching is $N(1-r)$, so the saving scales directly with the repeat rate — unlike model inference caching, where the value also depends on tail latency, embedding caching value is close to linear in $r$ because embeddings are typically used in bulk, offline pipelines rather than single interactive requests.

For a document corpus of size $D$ re-indexed on a schedule, a content-hash cache reduces re-indexing cost from $O(D)$ embedding calls per run to $O(\Delta)$, where $\Delta$ is the number of documents that actually changed since the last run — the mechanism underlying incremental indexing pipelines.

Cache key design as a correctness boundary

The cache key is a claim about everything that could change the output. For an embedding cache, that includes at minimum: the exact text, the model identifier and version, and any preprocessing applied before embedding (truncation length, normalisation, prompt templates some embedding models require). Omitting any of these from the key does not raise an error. It silently returns a stale or wrong vector — which is why embedding-cache bugs tend to surface as slowly degrading search quality, rather than crashes.

Storage layout trade-offs

  • Row-per-vector (as in the demo) — simple, supports point lookups by key, poor for bulk similarity search without an additional index.
  • Columnar / memory-mapped arrays (.npy, Arrow, Parquet) — better for loading an entire embedding matrix for batch similarity search, worse for single-key point updates.
  • Vector databases (FAISS, Milvus, pgvector) — combine storage with an approximate nearest-neighbour index, at the cost of an additional moving part; appropriate once lookup, not storage, is the bottleneck (see approximate nearest-neighbour search).

Quantised storage

Storing embeddings as float32 is often unnecessary precision. Embedding quantisation — storing vectors as int8 or even binary — trades a small, usually acceptable loss in retrieval accuracy. In exchange it gives a 4x to 32x reduction in storage and memory bandwidth, which matters once a cache holds millions of vectors. See storing vectors as int8 and binary for the mechanics and the accuracy trade-off in detail.

What to learn next