Learn AI Create the Future — with Pranay
Main site

Feature and Data Pipelines in Production

Low-latency feature lookup

Low-latency lookup means finding one entity's stored features in about a millisecond, using an index built ahead of time instead of searching row by row.

Read these first

On this page 7
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Low-latency lookup means finding one entity's stored features in about a millisecond, using an index built ahead of time.

The analogy you have already lived

A librarian does not walk every shelf to find your book. She checks the card catalogue first — a small index at the front, sorted, built in advance. It points straight to the right shelf.

Searching shelf by shelf works. It does not scale. And a model waiting on an answer has no time to spare for a librarian who skipped the catalogue.

Why it exists

Model serving usually has a budget of tens of milliseconds for the whole request. Most of that budget is gone before the model itself even runs — spent fetching the features it needs.

Fetching a user's features can mean scanning a table of millions of rows looking for a match. That scan alone can blow the entire budget. A lookup needs to go straight to the right row instead. That is the same way a catalogue card sends you straight to the right shelf.

How it works

slow:  scan every row, checking "is this the user I want?"
       -- gets slower as the table grows

fast:  look up the user's ID in an index built ahead of time
       -- index points straight at the row
       -- same speed no matter how big the table gets

The index is usually a key-value store — Redis, DynamoDB, or a similar system. It is built and refreshed by the feature pipeline, and sits right next to the model server for a short network hop.

A real example you have seen

Your phone's contacts app finds a name the instant you start typing. It is not reading every contact in order — it is using an index built when the contacts were saved.

Remember this

  • A live model has a tiny time budget, and feature lookup eats into it before the model even runs.
  • Scanning a table gets slower as it grows. An index built ahead of time does not.
  • The index lives close to the model server, refreshed by the pipeline ahead of the request, never computed on demand.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install pandas

Scan versus index, timed

lookup_speed.py
import time
import random
import pandas as pd

random.seed(0)
N = 200_000
table = pd.DataFrame({
    "user_id": range(N),
    "risk_score": [round(random.random(), 3) for _ in range(N)],
})

# Approach A: scan the table with a filter, like querying a big table
# fresh on every request.
def lookup_scan(user_id):
    row = table[table["user_id"] == user_id]
    return None if row.empty else row["risk_score"].values[0]

# Approach B: an index built once, ahead of time -- a plain dict here,
# standing in for a key-value store like Redis in front of the model.
index = dict(zip(table["user_id"], table["risk_score"]))

def lookup_index(user_id):
    return index.get(user_id)

targets = [random.randint(0, N - 1) for _ in range(200)]

t0 = time.perf_counter()
for uid in targets:
    lookup_scan(uid)
scan_time = time.perf_counter() - t0

t0 = time.perf_counter()
for uid in targets:
    lookup_index(uid)
index_time = time.perf_counter() - t0

print(f"table rows           : {N}")
print(f"lookups timed        : {len(targets)}")
print(f"scan lookup   total  : {scan_time*1000:.1f} ms  ({scan_time/len(targets)*1000:.3f} ms/lookup)")
print(f"indexed lookup total : {index_time*1000:.1f} ms  ({index_time/len(targets)*1000:.4f} ms/lookup)")
print(f"speedup              : {scan_time/index_time:.0f}x")
Output
table rows           : 200000
lookups timed        : 200
scan lookup   total  : 35.9 ms  (0.180 ms/lookup)
indexed lookup total : 0.1 ms  (0.0005 ms/lookup)
speedup              : 355x

This ratio came from one run on one machine with 200,000 rows — treat the 355x as illustrative, not a promise. What holds regardless of machine: the scan's per-lookup cost grows with table size, and the indexed lookup's does not.

Line-by-line walkthrough

lookup_scan filters the whole DataFrame every call — pandas has to check every row against the condition, even though only one row can ever match.

lookup_index builds a Python dict once, up front. A dict lookup by key does not depend on how many other keys exist — that is the entire reason it is fast, and the entire reason a real feature store's online layer is a key-value store, not a table you filter.

Common mistakes

Querying a data warehouse (BigQuery, Snowflake) directly from the serving path. These are built for large scans over huge data, not single-row point lookups under a millisecond. Use them to build the index; never query them per-request.

Rebuilding the index on every request instead of once, kept warm. This throws away exactly the advantage an index provides. Build it in the background, refresh it on a schedule or on new events, and read from the already-built copy.

No fallback for a missing key. A new user with no row in the index is not an error — it is expected. Decide the fallback deliberately; see missing features at serve time next.

Try it yourself

Grow N to 2,000,000 and re-run. Time the scan again. The indexed lookup's time per call should barely move — confirm that from your own numbers rather than trusting this page's.

What to learn next

Researcher — Mathematics and papers.

Complexity

A scan against an unindexed table costs $O(n)$ per lookup, where $n$ is the table size. A hash-based index costs $O(1)$ expected time per lookup, independent of $n$, at the cost of $O(n)$ space to build and maintain the index itself. This is the standard space-time trade-off underlying every key-value store used for online feature serving.

What production online stores actually add

A Python dict demonstrates the complexity argument; it is not what runs in production. Real online feature stores (Redis, DynamoDB, ScyllaDB, Bigtable) add three things a plain hash map does not provide:

  • Persistence and replication — surviving a process restart without losing the index, and serving reads from multiple replicas.
  • TTL and eviction — expiring stale entries automatically, tying directly into feature freshness.
  • Network-accessible access with bounded tail latency — Redis's single-threaded event loop gives p99 latencies in the low single-digit milliseconds for simple GETs at moderate load, but degrades under large values or expensive commands, which is why feature values are kept small and flat rather than nested objects.

The dual-store pattern

Nearly every production feature platform separates two stores with different access patterns:

  • Offline store — a data warehouse or lake (BigQuery, Snowflake, Parquet on S3), optimised for large scans, used for training set generation and backfills.
  • Online store — a key-value store, optimised for single-key point lookups under a strict latency budget, used only at serving time.

The feature pipeline's job is materialising the same feature definitions into both, keeping them consistent — the training-serving skew concern already raised in backfilling. Feast, Tecton and Databricks Feature Store are all organised around this exact split.

Papers and systems

  • DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store, SOSP 2007 — the foundational design for the online store category.
  • Chang et al., Bigtable: A Distributed Storage System for Structured Data, OSDI 2006.
  • Hermann and Del Balso (Meta), on the offline/online feature store split as a production necessity, not an implementation detail.

What to learn next