Section 7 · Learn · Generation
Constrained decoding and structured output
Forcing a model's output to match a schema during generation by masking invalid tokens, rather than validating and retrying afterwards.
Forcing a model's output to match a schema during generation by masking invalid tokens, rather than validating and retrying afterwards.
What it is
A grammar compiled from the schema restricts which tokens may be sampled at each step, so the output cannot leave the grammar.
Why it is needed
A retry loop wastes tokens and latency and can loop forever. Constraining at the sampler makes malformed output impossible rather than unlikely.
How it works
The schema is compiled to a state machine. At each decoding step, tokens that would leave the grammar are masked to negative infinity before sampling.
In this project
vLLM 0.27.1 supports xgrammar, guidance, outlines and lm-format-enforcer, with enforcement at the sampler rather than a retry loop. Three findings shaped the design: the backend is process-wide and not per-request; auto silently cascades xgrammar to guidance to outlines, so two runs could use different decoders with no signal in the output, and the backend should be pinned explicitly; and constrained decoding guarantees grammar conformance, not full JSON Schema validity.
What it does not guarantee
Keywords a grammar cannot express — numeric minimum and maximum, minLength, minItems — are not enforced at the sampler. The output is guaranteed to parse and to match the structural grammar. It is not guaranteed to satisfy every assertion in the schema. The design consequence is to keep the contract structural and put semantic assertions in a validation pass afterwards.
Thinking tokens are charged against the output budget
On models that reason before answering, thought tokens count toward maxOutputTokens. The budget runs out mid-JSON and the response arrives truncated but parseable up to the cut, with only finishReason indicating anything is wrong. Pinning the thinking level low removed the truncation and halved latency.
Trade-offs
Advantages
- Malformed output is impossible, not merely unlikely
- No retry loop
- Lower latency than validate-and-retry
Limitations
- Grammar compilation has a cost
- Not all schema keywords are expressible
- Over-constraining can degrade answer quality
Common mistakes
- Assuming schema-valid means semantically valid
- Leaving the backend on auto and getting a silent fallback
- Ignoring finishReason and parsing truncated JSON into a real-looking record
When to use it
Reach for it when
- Downstream code depends on the shape of the output
Avoid it when
- Free-form prose
How it interacts with the rest of the system
Citation resolution and quote verificationModel pinning and reproducibilityRetrieval-Augmented Generation