Observability for LLM Applications
Tracing an LLM application
A trace is the full step-by-step record of everything a request touched on its way to an answer, and it is the one thing you need before any other LLM observability technique makes sense.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 8
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
A trace is the complete, step-by-step record of everything a request touched on its way to an answer.
The analogy you have already lived
Ordering something online shows a tracking page. Order placed. Picked up. At the sorting hub. Out for delivery. Delivered.
Each line is a stop the parcel actually made, with a time attached. When a parcel goes missing, that list is exactly where you look. It shows the last place it was seen.
A trace is that same tracking page, for one request into an app built on a language model.
Why it exists
A single answer from an LLM app is rarely one step. It is usually several: look something up, build a prompt, call the model, check the reply, maybe call the model again.
When an answer comes back wrong, "which of those steps caused it?" is the entire question. Without a trace, that question has no evidence to work from.
How it works
a user's question
|
v
[ retrieve ] ----> 9 ms
|
v
[ build prompt ] ----> 0 ms
|
v
[ call the model ] ----> 51 ms <-- most of the time went here
|
v
[ check the reply ] ----> 0 ms
|
v
answer sent backEach named box is a span — one step, with a start time and a duration. A trace is the full set of spans for one request, strung together.
A real example you have seen
A support chatbot gives a confusing answer. The team behind it does not read the whole conversation and guess. They open that one request's trace, and see it retrieved the wrong document in the very first step. Everything downstream was doomed the moment that step ran.
The honest part
A trace tells you where time went and what each step saw. It does not automatically tell you why the model said what it said.
That second question is genuinely harder, and later lessons in this section build toward it. Tracing is the floor everything else stands on, not the whole answer by itself.
Remember this
- A span is one step; a trace is the full chain of spans for one request.
- Tracing answers "where did this request spend its time, and what did each step see?"
- It is the foundation every other LLM observability technique in this section depends on.
What to learn next
- Token and cost telemetry — the next thing worth attaching to every span built above.
- Redacting personal data from LLM logs — what to do before a trace's prompts and responses are stored anywhere.
- Model serving — the service whose requests this tracing wraps around.
Developer — Code and libraries.
Setup
No install needed. This script only uses Python's standard library.
A small tracer, from scratch
Enough to show the idea in full: a context manager that records a span's name, depth, and duration into a list, nested spans included.
import time
import uuid
from contextlib import contextmanager
TRACE = []
_stack = []
_t0 = time.perf_counter()
@contextmanager
def span(name, **attributes):
span_id = str(uuid.uuid4())[:8]
parent_id = _stack[-1] if _stack else None
start = time.perf_counter()
depth = len(_stack)
_stack.append(span_id)
try:
yield
finally:
_stack.pop()
duration_ms = (time.perf_counter() - start) * 1000
TRACE.append({
"id": span_id, "parent": parent_id, "name": name,
"start_ms": (start - _t0) * 1000, "duration_ms": duration_ms,
"depth": depth, **attributes,
})
def fake_retrieve(query):
time.sleep(0.01)
return [f"doc about '{query}', chunk {i}" for i in range(3)]
def fake_llm_call(prompt):
time.sleep(0.05) # standing in for a real network call to a model
return "Model serving means putting your trained model behind an address."
def answer_question(query):
with span("handle_request", query=query):
with span("retrieve"):
docs = fake_retrieve(query)
with span("build_prompt"):
prompt = "Context:\n" + "\n".join(docs) + f"\n\nQuestion: {query}"
with span("call_llm", model="fake-llm-v1"):
answer = fake_llm_call(prompt)
with span("parse_output"):
cleaned = answer.strip()
return cleaned
result = answer_question("what is model serving?")
print("answer:", result)
print()
TRACE.sort(key=lambda r: r["start_ms"])
print(f"{'span':<28}{'duration_ms':<14}attributes")
for row in TRACE:
attrs = {k: v for k, v in row.items()
if k not in ("id", "parent", "name", "start_ms", "duration_ms", "depth")}
indent = " " * row["depth"]
label = f"{indent}{row['name']}"
print(f"{label:<28}{row['duration_ms']:<14.2f}{attrs}")answer: Model serving means putting your trained model behind an address.
span duration_ms attributes
handle_request 60.68 {'query': 'what is model serving?'}
retrieve 9.86 {}
build_prompt 0.00 {}
call_llm 50.72 {'model': 'fake-llm-v1'}
parse_output 0.00 {}The content of the answer is fixed and exact. The duration_ms values are real timings from this run on this machine, and will differ every time you run it — including on a rerun on the same laptop. retrieve and call_llm use time.sleep() to stand in for real network latency; a real retrieval step or a real model API call would show its own, genuinely variable timing here instead.
Line-by-line walkthrough
_stack tracks which span is currently open, so a nested span can record its parent's ID. This is what turns a flat list into a tree.
fake_retrieve and fake_llm_call are stand-ins. Swap fake_llm_call for a real API call, and the tracer around it needs no changes at all.
The with span(...): blocks are the entire integration cost. Wrapping an existing function in one line of context manager is usually enough to start tracing it.
Common mistakes
Tracing only the call to the model, and nothing else. The slowest or most broken step is often retrieval or prompt construction, not the model call. Trace every step, not only the obvious one.
Not recording attributes alongside timing. A span with a duration and nothing else cannot tell you which query, which model version, or which documents were retrieved. Attach the details you would want while debugging at 2 a.m.
Building a custom tracer for a production system, past a prototype. This lesson's tracer is for learning the concept. For anything real, use an existing standard — OpenTelemetry, or an LLM-specific tool like LangSmith or Langfuse — so traces from different parts of a system stay comparable.
Logging the full prompt and response as plain text with no redaction. These often contain personal data. Redacting personal data from LLM logs covers this directly, and belongs in the same pipeline as tracing, not as an afterthought.
Try it yourself
Add a second call_llm span after the first, simulating a follow-up call the app makes to double-check its own answer. Rerun, and check how the trace shows two separate model calls inside one request.
What to learn next
- Token and cost telemetry — the next thing worth attaching to every span built above.
- Redacting personal data from LLM logs — what to do before a trace's prompts and responses are stored anywhere.
- Model serving — the service whose requests this tracing wraps around.
Researcher — Mathematics and papers.
The standard this converges toward
Production systems do not hand-roll a tracer like the one above. OpenTelemetry defines a vendor-neutral data model for exactly this: a Span with a trace ID, span ID, parent span ID, start and end timestamps, attributes, and events, propagated across process and network boundaries via context propagation headers (traceparent).
The LLM-specific layer on top of this (OpenTelemetry's GenAI semantic conventions, and tools like LangSmith, Langfuse, and Arize Phoenix) standardises attribute names for LLM-specific concepts: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.prompt, gen_ai.completion — so traces from different frameworks and providers remain comparable in one dashboard.
Distributed tracing across process boundaries
The single-process stack used in the developer example does not survive a request crossing into a second service (a separate retrieval service, a separate tool-execution service). Distributed tracing solves this by propagating a trace_id in request headers, so spans recorded in different processes, potentially on different machines, can still be reassembled into one tree by a collector. This is the same mechanism used for tracing in general microservice architectures (Sigelman et al., 2010, the Dapper paper), applied to an LLM pipeline's steps instead of RPC calls.
Sampling at scale
Recording every span for every request is $O(n \cdot s)$ in storage for $n$ requests with $s$ spans each, which becomes expensive at high volume for verbose LLM payloads. Production tracing systems apply sampling: head-based sampling decides whether to trace a request at its start (cheap, but risks missing a rare failure), while tail-based sampling buffers a request's full trace and decides whether to keep it after seeing the outcome (catches errors and outliers reliably, at higher buffering cost).
Papers and standards
- Sigelman et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google Technical Report 2010 — research.google/pubs/pub36356
- OpenTelemetry, Semantic Conventions for Generative AI Systems — opentelemetry.io/docs/specs/semconv/gen-ai
- Fonseca et al., X-Trace: A Pervasive Network Tracing Framework, NSDI 2007 — an earlier distributed-tracing design with a similar causal-tree model.
What to learn next
- Token and cost telemetry — the next thing worth attaching to every span built above.
- Redacting personal data from LLM logs — what to do before a trace's prompts and responses are stored anywhere.
- Model serving — the service whose requests this tracing wraps around.