An LLM response may look like one continuous stream of text, but the serving system handles it in distinct stages. Understanding those stages makes latency, GPU memory, and capacity decisions much easier to reason about.
1. From text to tokens
The request first passes through a tokenizer, which converts the prompt into token IDs understood by the model. A scheduler then decides when the request can run and which other requests can share the same GPU batch. Queue time is therefore part of user latency even before model computation begins.
2. Prefill and the first token
During prefill, the model processes all input tokens in parallel. At every transformer layer it computes the attention state for the prompt and stores the resulting keys and values. The hidden state at the final input position is projected into vocabulary scores, then the sampling strategy selects the first output token. Time to First Token includes scheduling, prefill, and this first sampling step, which is why long prompts usually take longer to start responding.
3. Why KV cache matters
Without KV cache, the model would recompute attention state for the entire prompt and every previous output token at each step. KV cache keeps the key and value tensors already produced at every layer. The next step computes only the new token and reads the earlier state from GPU memory. This saves substantial computation, but the cache grows with context length and often becomes the main limit on concurrent requests.
Serving engines commonly manage this memory in pages rather than reserving one large continuous block per request. Paged allocation reduces fragmentation and allows GPU memory to be shared more efficiently across short and long requests.
4. Decode generates the remaining response
After the first token, the model enters the decode loop. Each iteration processes the latest token, appends its key and value to the cache, attends to the existing context, and selects one new token. The loop continues until the model returns an end token, reaches the requested output limit, or fills the context window.
Decode handles only one new token per request at a time, but repeatedly reads model weights and an expanding KV cache. It is often limited by memory bandwidth rather than raw computation. Batching multiple requests helps the GPU reuse each model weight load across more tokens and improves throughput.
5. Prefix cache can skip repeated work
If many requests share the same system prompt, instructions, or document prefix, a serving engine can retain the corresponding KV cache and reuse it. Only the unmatched input tokens need prefill computation. Prefix caching can reduce Time to First Token significantly for repetitive enterprise workflows, although it does not remove the decode work for the new response.
The three metrics to watch
Time to First Token reflects queueing and prefill. Time per Output Token reflects decode speed. End to end latency combines both with the requested response length. A serving optimization should identify which stage limits the actual workload before changing models, hardware, or infrastructure.