AI infrastructure
What happens after an LLM request enters an inference system?
English adaptation of an earlier Chinese essay.
A chat interface makes the interaction look simple: send a question and receive an answer. Behind it, a serving system coordinates computation, memory and many requests sharing the same hardware.
Here is a high-level view of a typical autoregressive LLM request. The details and ordering around routing and tokenization vary by deployment.
Accept, route and tokenize
The service accepts the request and directs it to a model instance. A tokenizer converts the input text into token IDs. Instructions and conversation history can be included along with the latest question; together, these form the context for generation.
A scheduler decides when the request can run alongside other requests, subject to available compute and memory. Under load, time spent waiting is part of the delay before an answer begins.
Prefill: process the context
During prefill, the model processes the prompt and builds attention keys and values for its tokens. Prompt tokens can be processed in parallel within a forward pass, while respecting the causal attention mask. Long prompts may be processed in chunks.
Keep the state in a KV cache
The keys and values are retained in a KV cache. Later generation steps reuse this state instead of recalculating those earlier keys and values. The cache takes memory, and more active requests or longer contexts increase the amount of state the system must manage.
Decode: generate the continuation
In ordinary autoregressive decoding, generation proceeds one token at a time. Each step uses the previous context, extends the cached state and selects a next token. Different requests can still share a batch even though the tokens within one request depend on earlier tokens.
Stream the result
With streaming enabled, the service sends output as it becomes available. The wait for the first token and the pace of later tokens describe different parts of the experience. Queueing, prompt processing and token generation all affect how responsive the service feels.
An answer is the result of both a model and a serving system. Understanding this path makes it easier to reason about latency, concurrency, memory and the tradeoffs behind a smooth user experience.