Streaming

Streaming delivers a model's output piece by piece as each token is generated, instead of waiting for the whole answer and sending it at once.

It is the difference between a dosa restaurant and a parcel service. The dosa arrives at your table the moment it is off the tawa, and you start eating while the next one cooks. The parcel service would hold your entire meal until every item was ready, then deliver it cold, together. Same kitchen speed — completely different experience of waiting.

The reason streaming matters so much for LLMs is that generation is genuinely slow and sequential: one token at a time, often 30-100 tokens per second. A 500-token answer means 5-15 seconds of total generation. Without streaming, the user watches a spinner for all of it; with streaming, the first words appear after a fraction of a second and the answer types itself out. Total time is identical — perceived latency collapses. This is why every chat product streams, and why the metric time-to-first-token is tracked separately from tokens-per-second.

Technically, APIs implement it with server-sent events (SSE): one HTTP response kept open, tokens arriving as small events. Client code changes from one response to a loop over deltas:

stream = client.chat.completions.create(model="gpt-4o-mini",
    messages=msgs, stream=True)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Design consequences worth knowing: parsing structured-output mid-stream needs care (the JSON is incomplete until the end), output guardrails must either buffer or scan incrementally, and cancelling a bad answer early actually saves money — you stop paying for tokens never generated.

Where to go next#

• Full lesson: vLLM
• Related terms: latency, token, inference, throughput