TL;DR
- A text response costs one prefill plus one decode step per output token. For long answers, decode is most of the wait.
- A decision needs only the scores prefill already computes, so the request returns before decode starts.
- The same idea works upstream: vLLM Semantic Router’s classifier turns reasoning off when a query does not need it, cutting tokens by 48.5% and latency by 47.1% on MMLU-Pro.
very generated token is one more pass through the model. A 200-token answer costs one prefill and 199 decode steps, and most of a serving stack exists to batch, cache and schedule that loop.
Many requests never needed it. Route this ticket, flag this message, pick the next tool: each has a handful of valid answers. In this post, I walk through where decode time goes, how decision models avoid it, and what the public benchmarks show.
Where does the time go in a text request?
A text request has three phases:
- Host preparation. The server applies the chat template, tokenizes the text into IDs on the CPU, and schedules the request into a batch.
- Prefill. All input positions run through the transformer once. The last position outputs logits, one score per vocabulary token. The first output token is picked from these.
- Decode. The picked token goes back into the model to predict the next one. The KV cache saves earlier keys and values, but each new token is still a full pass through every layer.
Prefill is paid once. Decode is paid per token. That is why long answers are slow, and why skipping decode is the biggest lever a decision has.
Where can a decision finish early?
At the logits. Text output is open-ended, so the server has to keep generating. A decision fixes the answers in advance, so the server only needs their scores.
On an ordinary chat model, label the options A, B and C, run prefill, and read the logits of those three tokens. A softmax turns them into probabilities. Nothing is fed back into the model. The scores are the output, so there is no next token to predict.
Two caveats:
- The prompt must match the readout. Scores after a prompt that asks for JSON are not the probabilities of a prompt that asks for A, B or C.
- The API can still return JSON. Server code serializes the answer. This is different from JSON-schema generation, which constrains tokens while decode keeps running.
What is a decision model?
A decision model maps unstructured input to a typed answer from a fixed set, with a probability for each option. TypeSafe calls these System One models: “unstructured state in, typed probabilistic decisions out.”
Three question types cover most uses:
- Choice: one of several options, such as billing, sales or support.
- Score: a point on a bounded scale, such as severity 0 to 9.
- Yes or no: a binary check, such as a policy violation.
The interface has standardized faster than the models behind it. SGLang’s decision endpoint, two vLLM prototypes and the CLM server all accept the same /v1/systemone request shape over very different architectures.
How do today’s decision models read the answer?
Five public approaches. They differ in what runs per request and what can be reused.
All speeds are self-reported on different hardware. Treat them as orders of magnitude, not a ranking.
Label scoring on a chat model
SGLang reads label log-probabilities from any chat model after prefill. Labels must be single tokens (A to Z, 0 to 9, yes or no), and each question gets its own prefill. Trained checkpoints such as Clef answer a full schema in one prefill.
Encoder with a decision head
Laya is “a ModernBERT/mmBERT encoder plus a decision head, not a generative model.” Options are marked with [MASK] tokens; one encoder pass scores every marker. The read is deterministic, so sampling settings have no effect.
Contrastive state and action scoring
CLM embeds the state and each action separately and scores them by vector alignment. A frozen Qwen3-8B backbone carries a ~20M-parameter projection head. Because states and actions are encoded apart, both are cacheable: a revisited state drops from 1.7 ms to 0.6 ms p50.
Answer slots on a diffusion canvas
DiffusionGemma works on a fixed-width canvas. The vLLM prototype seeds a template with blank answer slots and reads all of them in one denoise step. Labels must be single tokens so the canvas never shifts. Uncertain slots get re-read.
Purpose-built System One model
TypeSafe describes Jev as a new architecture with a parallel sampler that “generates all outputs in a single query,” trained for calibrated probabilities. Its internals are not public, so there is no diagram here.
Can a decision model speed up a text model?
Yes, by deciding how much decoding a request deserves. vLLM Semantic Router runs a ModernBERT classifier, in Rust with Candle, in front of vLLM. It reads each query and decides whether it needs chain-of-thought reasoning. Simple queries go to a fast path; hard ones get reasoning.
The classifier is itself a decision model: one encoder pass, no decode. What it saves is the generator’s decode. On MMLU-Pro across 14 domains, with Qwen3-30B-A3B on vLLM v0.10.1 and NVIDIA L4 GPUs, the authors of When to Reason: Semantic Router for vLLM report:
Latency falls almost exactly in step with tokens. Halve the decode work and the wait halves with it. Accuracy rises because hard questions still get reasoning; business and economics gained more than 20 points.
The paper does not report the classifier’s own latency. The router’s design target is under 100 ms for all routing signals combined, small next to a 13-second answer.
Where does the speed come from?
Six levers, shared across all five approaches:
- No decode loop. Everything after prefill shrinks to a gather and a softmax.
- Many questions, one pass. Clef answers a schema in one prefill; Laya answers six questions in a 14 ms request. Plain label scoring pays one prefill per question.
- A narrower output. Reading three label logits is cheaper than scoring a full vocabulary. Modal’s Quail trims its output head to eight tokens at startup.
- Reuse. CLM caches state and action vectors. Prefix caching does the same for shared prompts on decoders.
- A smaller backbone. Laya runs a BERT-class encoder. CLM trains only a 20M-parameter head.
- Less state. No generated tokens means no growing KV cache, so memory frees sooner.
What does not get faster: tokenization, attention over a long input, and the extra input that more options add.
