Semantic caching
A semantic cache recognises when a new question means the same thing as an old one, even in different words, and reuses the old answer.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 8
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
A semantic cache reuses an old answer when a new question means the same thing. It works even if the words are different.
The analogy you have already lived
A shopkeeper who has worked the same counter for ten years does not need your exact words. You ask for "the red pen," or "a ballpoint, red one." Or you point and say "that one." They hand you the same pen every time.
They are matching what you mean, not the exact sentence you used. The plain cache from the last lesson is a shopkeeper who only recognises your exact words. Phrase it differently, and it makes you repeat the request from scratch.
Why it exists
The cache from the last lesson only fires on an exact match. A person reads "How do I reset my password?" and "how can I reset my password" as identical. A computer comparing text character by character sees them as completely different.
Real users ask the same question in dozens of ways. A cache that only catches exact repeats misses almost all of them. The model ends up re-answering the same question over and over, in slightly different clothes.
How it works
new question arrives
|
v
turn it into a list of numbers that capture its MEANING
(this list is called an "embedding")
|
v
compare those numbers to the numbers of every cached question
|
v
close enough? ---- yes ----> reuse the old answer
|
no
v
ask the model for real, then save this question's numbers tooAn embedding is a list of numbers a model produces for a piece of text. It is built so texts with similar meaning end up with similar numbers. "How do I reset my password?" and "I forgot my password" land close together. "How do I reset my password?" and "what's the weather" land far apart.
A real example you have seen
Customer support chatbots that answer instantly even when you phrase your problem in your own words. Search boxes that find the right help article even when you did not use its exact title. Both are usually built on this idea: match meaning, not exact text.
The honest part
"Close enough" is a judgement call, not a fact. Set the bar too loose, and the cache confidently answers a different question with an old, wrong answer. Set it too strict, and you get almost no benefit over exact matching.
There is no universal correct setting. It depends on how costly a wrong answer is for your particular use case. That is worth deciding on purpose, not by accident.
Remember this
- A semantic cache matches meaning, using embeddings, not exact text.
- It catches paraphrases that an exact-match cache misses completely.
- The similarity threshold is a real trade-off between more savings and more wrong answers.
What to learn next
- Prompt prefix caching — reusing computation inside one long prompt, not just across separate questions.
- Caching embeddings — the embeddings this cache computes are themselves worth reusing.
- FAISS — the tool to reach for once a linear scan stops being fast enough.
Developer — Code and libraries.
Setup
pip install sentence-transformers numpyA cache that matches meaning
import time
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
def slow_llm_call(question: str) -> str:
time.sleep(0.05) # pretend this is a real, paid LLM call
return f"[real answer for: {question!r}]"
class SemanticCache:
def __init__(self, threshold=0.85):
self.threshold = threshold
self.questions, self.answers, self.vectors = [], [], None
def get(self, question: str):
q_vec = model.encode([question], normalize_embeddings=True)[0]
best_sim, best_idx = 0.0, -1
if self.vectors is not None:
sims = self.vectors @ q_vec # cosine similarity; vectors are already normalised
best_idx = int(np.argmax(sims))
best_sim = float(sims[best_idx])
if best_sim >= self.threshold:
return "HIT", best_sim, self.questions[best_idx], self.answers[best_idx]
answer = slow_llm_call(question) # real work happens only here
self.questions.append(question)
self.answers.append(answer)
new_vec = q_vec.reshape(1, -1)
self.vectors = new_vec if self.vectors is None else np.vstack([self.vectors, new_vec])
return "MISS", best_sim, None, answer
cache = SemanticCache(threshold=0.85)
queries = [
"How do I reset my password?",
"What's the weather like today?",
"I forgot my password, how do I change it?",
"Explain how I can change my forgotten password",
]
for q in queries:
start = time.perf_counter()
status, sim, matched, answer = cache.get(q)
ms = (time.perf_counter() - start) * 1000
tag = f"matched {matched!r}" if matched else "no match close enough"
print(f"{status:4} sim={sim:.2f} {ms:6.1f} ms {tag}")MISS sim=0.00 275.8 ms no match close enough MISS sim=0.00 60.2 ms no match close enough HIT sim=0.90 10.5 ms matched 'How do I reset my password?' MISS sim=0.83 58.6 ms no match close enough
Timings are real, from this machine, and include the embedding model's own compute. Yours will differ. The first call is slower because the model finishes warming up on it. Read the pattern, not the exact milliseconds.
Line-by-line walkthrough
normalize_embeddings=True. This scales every embedding to length 1. Once that is true, the dot product between two vectors is their cosine similarity. So self.vectors @ q_vec computes every similarity in one matrix multiply.
self.threshold = 0.85. This is the "close enough" line. The output above shows why it matters. A genuine paraphrase scored 0.90 and hit; a more loosely related rephrasing scored 0.83 and missed. Move that line by 0.05 in either direction and both outcomes flip.
The linear scan (self.vectors @ q_vec). This checks every cached question, one matrix multiply. It is fine for hundreds or low thousands of entries. For more, see FAISS and approximate nearest-neighbour search — the same idea, built to stay fast at millions of entries.
Common mistakes
Trusting one threshold forever. A threshold tuned on today's traffic can start misfiring as the questions users ask change. Recheck it periodically against real examples of hits and misses.
Caching questions with a hidden extra input. "What's my balance?" means something different for every user. If the user's identity is missing from the cache key or the embedded text, one person's cached answer can leak to another.
Never expiring anything. A semantic cache still needs the same expiry discipline as an exact-match one. See the mistakes in caching model predictions. Meaning-matching does not make a stale answer less stale.
Skipping normalisation. Forgetting normalize_embeddings=True, or dividing by the vector length yourself, turns the dot product into something that is no longer a clean similarity score. Thresholds tuned assuming cosine similarity stop meaning what you think they mean.
Try it yourself
Print best_sim for every query, hit or miss, and plot it. Look for the gap between your genuine paraphrases and your genuinely different questions. That gap is where your threshold should sit.
What to learn next
- Prompt prefix caching — reusing computation inside one long prompt, not just across separate questions.
- Caching embeddings — the embeddings this cache computes are themselves worth reusing.
- FAISS — the tool to reach for once a linear scan stops being fast enough.
Researcher — Mathematics and papers.
Cosine similarity and the threshold trade-off
For two embeddings $u, v \in \mathbb{R}^d$, cosine similarity is
$$\text{sim}(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert} \in [-1, 1]$$
When both vectors are pre-normalised to unit length, this reduces to the plain dot product, which is why the demo code normalises embeddings before storing them — it turns a similarity search into a single matrix multiply with no per-pair division.
The threshold $\tau$ trades two error types against each other:
- False hit (semantic drift): $\text{sim}(u,v) \geq \tau$ for questions that are not actually equivalent. Costs correctness — a wrong cached answer is served with full confidence.
- False miss: $\text{sim}(u,v) < \tau$ for questions that are equivalent. Costs money — the cache should have fired and did not.
There is no threshold-free way to avoid both. Vendor SDKs for semantic caches (Redis, GPTCache and similar systems) commonly default $\tau$ in the 0.8–0.95 range, but the correct value is a property of your embedding model and your traffic, not a constant to copy from documentation.
Consistency guarantees
An exact-match cache is correct by construction: identical keys imply identical prior computation. A semantic cache gives up that guarantee — it substitutes approximate equivalence for exact equivalence, and inherits every property of the embedding model, including its blind spots. Negation is the sharpest example. "how do I enable notifications" and "how do I disable notifications" embed close together under most sentence encoders, because the surrounding words are nearly identical — while the correct answers are opposites. A semantic cache with too generous a threshold will confidently merge these.
Retrieval at scale
The demo's linear scan is $O(n)$ per lookup. Past low thousands of cached entries, replace it with an approximate nearest-neighbour index — HNSW or IVF, covered in approximate nearest-neighbour search — which trades a small, tunable amount of recall for sublinear lookup time.
Papers and systems
- Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019 — arxiv.org/abs/1908.10084
- Bang, GPTCache: An Open-Source Semantic Cache for LLM Applications, 2023 — github.com/zilliztech/GPTCache
What to learn next
- Prompt prefix caching — reusing computation inside one long prompt, not just across separate questions.
- Caching embeddings — the embeddings this cache computes are themselves worth reusing.
- FAISS — the tool to reach for once a linear scan stops being fast enough.