Same model, 15 providers, 84.63% to 100% tool-call accuracy
Two rows out of a table Moonshot publishes in their K2 Vendor Verifier repo:
| Provider | schema_accuracy | successful / triggered |
| --- | --- | --- |
| Fireworks | 100.00% | 1703 / 1703 |
| Together | 84.63% | 1602 / 1893 |
Same weights. Same 4,000-request test set. Same reference to compare against (Moonshot's own endpoint decides what a correct response looks like). The only variable is who served the model.
Moonshot built that harness after a run of "K2 is bad at tool calling" reports that turned out to be deployment problems. Their write-up is direct about why: the more open the weights, the more channels they get deployed through, and the less control anyone has over the result. From the outside you can't tell a model defect from an engineering deviation, and if users can't tell, the model gets blamed for both.
The two metrics measure different failures
tool_call_f1 scores the decision to call a tool at all. It's computed off finish_reason: true positive when the deployment and the reference both return tool_calls, false positive when only the deployment does, false negative when only the reference does. It answers one question. Does this endpoint decide to call tools at the same moments the official API does.
schema_accuracy is count_successful_tool_call / count_finish_reason_tool_calls. It only counts responses that already committed to a tool call, then asks how many produced arguments that validate against the JSON schema.
The second metric is the one that reaches your error handler. finish_reason says tool_calls, your dispatcher pulls the arguments out, the payload doesn't match the schema, and you get something like this in the middle of a 40-step run:
jsonschema.exceptions.ValidationError: 'path' is a required property
at tools.read_file(arguments={'file': 'src/api/handlers.ts'})
A retry sometimes papers over it. It costs you a round trip and a chunk of context either way, and in a loop that is already deep into a plan, the retry happens without the model knowing why.
The spread, on identical weights
Kimi K2 Thinking, 15 deployments, temperature 1.0:
| Provider | schema_accuracy | successful / triggered |
| --- | --- | --- |
| MoonshotAI (official) | 100.00% | 1958 / 1958 |
| Fireworks | 100.00% | 1703 / 1703 |
| InfiniAI | 99.89% | 1825 / 1827 |
| SiliconFlow | 98.96% | 2097 / 2119 |
| GMICloud | 95.95% | 1775 / 1850 |
| AtlasCloud | 95.74% | 1798 / 1878 |
| SGLang | 95.52% | 1790 / 1874 |
| vLLM | 87.22% | 1856 / 2128 |
| Parasail | 87.14% | 1837 / 2108 |
| DeepInfra | 86.91% | 1800 / 2071 |
| GoogleVertex | 85.76% | 1668 / 1945 |
| Together | 84.63% | 1602 / 1893 |
| Chutes | 83.05% | 3037 / 3657 |
Sixteen points between the top and the bottom of the same model. On the K2 0905 table the floor is lower: five deployments at 100%, then vLLM at 76.00%, SGLang at 73.13%, Volc at 72.86%, Baseten at 72.49%, Together at 71.96%. Roughly 28 points of spread on identical weights.
A perfect schema score can still be a broken deployment
Look at Groq on the 0905 table: 100.00% schema accuracy, and a trigger F1 of 69.52%. The schema metric only looks at calls the model already decided to make, so a deployment that fires tool calls at the wrong moments keeps a clean schema score. Nebius shows the same shape, 84.47% schema with 50.60% F1. Read the pair together or the number lies to you. High schema plus low F1 means the endpoint is confident about when to call tools and wrong about it.
Per-call accuracy is the wrong unit
Schema accuracy is a per-call number. Your agent doesn't make one call, it makes a session's worth, and every call has to validate. If the per-call probability is p, the chance that all N calls in a session validate is p^N.
| Provider | per call | 10 calls | 25 calls |
| --- | --- | --- | --- |
| Fireworks | 100.00% | 100.0% | 100.0% |
| SiliconFlow | 98.96% | 90.1% | 77.0% |
| GMICloud | 95.95% | 66.1% | 35.6% |
| vLLM | 87.22% | 25.5% | 3.3% |
| Together | 84.63% | 18.9% | 1.5% |
| Chutes | 83.05% | 15.6% | 1.0% |
A 1.04 point gap between SiliconFlow and Fireworks looks like noise per call. Over a 25-call session it's a 23 point gap. This is the same reason a cost-per-PR number is a distribution rather than a price: the interesting behavior lives in the tail of the repeated event, not in the average of one.
Two honest caveats on the table above. The denominators differ per provider (Moonshot's official run triggered 1958 tool calls, Together's 1893, Chutes' 3657), so these are not perfectly paired samples. And a schema failure in production is often a retry rather than a dead run, so p^N is a lower bound on session success, not a prediction of crash rate. The shape holds either way: small per-call differences get amplified by repetition, and the amplification is where agents live.
The same provider shows up at both ends
DeepInfra is 100.00% on k2-0905-preview at temperature 0.6, and 86.91% on k2-thinking at temperature 1.0. Together is 71.96% on one table and 84.63% on the other. So a provider's score is a property of a specific model version, decoding parameters, and serving stack, not a brand fact you can carry forward. KVV's first fix after the initial reports was exactly this class of thing: enforcing Temperature=1.0 and TopP=0.95 in thinking mode, and requiring that thinking content be passed back to the API.
The repo also names two failure modes that are worth reading before you file a bug against a model. Tool call IDs have to follow the format functions.func_name:idx, and Moonshot's own API rewrites incoming IDs into that shape automatically. A provider that doesn't rewrite them feeds the model malformed IDs from history and gets parsing failures back. And there's no built-in schema enforcement: the model generates tokens by probability, so the README tells vendors to add guided encoding rather than trust prompting. Their recommended versions are vLLM v0.11.0 or newer and SGLang v0.5.3rc0 or newer for 0905, with v0.11.1rc6 and v0.5.5.post2 for thinking.
If you route through an aggregator, you're already measuring a moving target
Most teams don't pin a provider. They point at a router and take whatever answers. OpenRouter's default is price-based load balancing across providers with allow_fallbacks set to true, so the endpoint that serves your request can change between runs without any config change on your side. Their docs are explicit that a sort or order preference disables load balancing.
That means your eval score, your p95 latency, and your schema accuracy are all attached to a provider you didn't choose. For an eval run, pin it:
extra_body={"provider": {"only": ["fireworks"], "allow_fallbacks": False}}
If you run a reviewer or an agent on your own keys, the provider is a config field you own rather than a default you inherit. Kodus's BYOK setup has that shape, keys in and a model choice per task, and it's the right shape for this problem: the provider is a variable you set per job and can hold constant while you measure something else.
A protocol you can run in an afternoon
1. Pick one model version and one decoding parameter set. Write both down, because the next table you compare against won't have them.
2. Run your own tool-call test set against at least two providers, and log finish_reason plus a schema validation result per response. Do not log only pass or fail on the session, or you'll lose the per-call number.
3. Report schema_accuracy next to trigger F1. A clean schema score on its own has already been shown to hide a deployment that calls tools at the wrong times.
4. Re-run after every version bump on either side, and treat a provider change as a model change for eval purposes.
The KVV table is a deployment check, not a model review. The useful takeaway isn't which vendor is on top this month, it's that identical weights produce a 16 to 28 point spread in the number your agent depends on, and that spread is something you can reproduce on your own endpoint in an afternoon. If you can't reproduce it, you don't have a number. You have a vendor's number.
vLLM Tool Calling Not Working? 1 in 12 Calls Fail by Default. Here's the Fix
If your vLLM tool calling is not working with Kimi or DeepSeek, and your agent keeps emitting broken JSON, inventing arguments or looping on the same call, check your server before you blame the model. Ubicloud measured Kimi K2.6 on vLLM 0.28 and got schema-valid tool calls only 92% of the time. That is 2 failures in every 25 calls, roughly 1 in 12.
With constrained decoding turned on, Ubicloud got to 100%. vLLM doesn't turn it on for you, even though Moonshot, who make the model, tell vendors to use it (they call it guided encoding). On vLLM 0.31+ the fix is one server flag, --tool-strict-level parameter, and the rest of this post is about why and how to check it worked.
A lot of the "open models are bad at agents" talk I've seen comes down to a server default.
What broken vLLM tool calls look like
On the client it rarely says "tool call failed". Instead you get:
- arguments that won't parse as JSON, or parse but miss a required field
- a parameter the tool never declared, or an enum value that isn't in the enum
- the agent reading back its own validation error, retrying with the same mistake and burning your token budget in a loop
- a framework that quietly drops the call and lets the model answer in prose
Each of these looks like a model problem, so people swap models or add three paragraphs of "ALWAYS RETURN VALID JSON" to the system prompt. Neither fixes it.
92% sounds fine until you chain calls
Per-call accuracy compounds. At 92%, a 10-step agent run gets through clean 0.92^10 ≈ 43% of the time. At 20 steps it's about 19%. In the Hacker News thread, one commenter worked out that even 95% per call gives you about an 8% chance of finishing a 50-call sequential task. Someone else in the same thread said they run around 300 tool calls per prompt.
92% wasn't a fluke of Ubicloud's setup, either. Moonshot's own K2 Vendor Verifier results (test date 2025-11-15) show the vLLM reference deployment at 87.22% schema accuracy on K2-thinking and 76.00% on K2-0905, while Moonshot's API and providers like Fireworks hit 100% serving the same weights.
Why vLLM leaves tool calls unconstrained by default
Without constraints, the model samples tool-call tokens like any other text. Kimi K2.6's format looks like this:
<|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>
Nothing stops the model from writing an unknown function name, a non-integer index or JSON that ignores your schema. Constrained decoding uses XGrammar structural tags to mask the vocabulary at each step, so once the model opens a tool call it can only produce a declared tool name, then an integer index, then JSON that matches that tool's schema, then the end tag.
vLLM supports this, just not on every request. PR #45600, merged in June 2026, changed tool_choice="auto" so structural tags apply only when at least one tool sets "strict": true. A missing strict counts as not strict. A reviewer on that PR objected that Claude Code's protocol doesn't include a strict field at all, which left Messages API clients with no obvious opt-in; the PR ended up adding strict to vLLM's Anthropic tool model. Plenty of OpenAI-style agent code doesn't set it either.
So if you're on tool_choice="auto" (most agents are) without strict on your tools, you're sampling free-form. Ubicloud originally added constrained decoding in their own vLLM fork. After vLLM shipped the feature in v0.25 they switched to upstream, and their score dropped back to 91% because of this default change.
The fix: --tool-strict-level in vLLM 0.31+
vLLM 0.31.0 added a server-side floor, --tool-strict-level (PR #56268). From the tool calling docs:
| Value | What gets constrained |
|---|---|
| auto | Default for most parsers. Structural tags only for required/named tool choice, or auto when a tool sets strict: true |
| function | Tool-call envelope (markup, function name) on every request with tools. Arguments stay free unless the tool is strict |
| parameter | Everything in function, plus argument schemas for every tool, as if all were strict |
If you don't set it, the parser's default applies: function for glm47, auto for everything else, Kimi included.
For a Kimi deployment I'd go straight to parameter. Invented args and missing required fields are argument problems, and function doesn't touch those:
vllm serve <your-kimi-k2.6-checkpoint> \
--enable-auto-tool-choice \
--tool-call-parser kimi_k2 \
--tool-strict-level parameter
Two things to know before you deploy it:
- It's a floor. It never loosens what a request already asked for, and it doesn't force a tool call. The grammar engages only after the model opens the tool-call wrapper itself.
- VLLM_ENFORCE_STRICT_TOOL_CALLING=false turns structural tags off entirely and overrides this flag. If someone set it while chasing a bug months ago, the flag does nothing.
Stuck below 0.31? Set "strict": true on every function definition you send. Under tool_choice="auto" that turns on the same structural tags without a server flag.
Where constrained decoding still bites
This has costs, and the HN thread argues about them at length.
A grammar guarantees valid syntax but can't make the arguments correct: a well-formed call to the wrong tool with a wrong URL still passes. One commenter argued that grammar masking only gives you token-level retries rather than retrying the whole sequence, so it can skew output away from what the model would have picked, and suggested just retrying failed calls. They later allowed that retries might not win once you count cost. Ubicloud's post calls constrained decoding "a per-model task", and one of its authors said in the thread that Qwen 3.8 doesn't come with the clear recommendation Moonshot gives for Kimi, and advised checking model quality with it on and off. Measure both.
It also has bugs you should know about:
- #58930: JSON schemas nested inside structural tags skip XGrammar's unsupported-feature check, so a constraint like multipleOf is silently ignored. Still open at the time of writing, with a fix proposed in PR #58940.
- #44006: strict tool calling plus MTP speculative decoding threw "Failed to advance FSM" errors and 500s. Closed by PR #44297 in July 2026, though a later comment on v0.28.0 still saw the FSM error, just without the 500.
- #55152: the guidance backend rejects the new structural tag format with KeyError: 'triggers'. A vLLM contributor replied that the tool-call structural tags were built for xgrammar and expects all of them to fail on guidance.
Ubicloud also notes that gpt-oss-20b, Qwen 3.6 and GLM 5.1 users reported issues back when constrained decoding was on by default. PR #45600 says it wanted to stop structural tags from over-constraining generation when the caller hadn't asked for strict. I think that was the wrong trade: a few loud regressions on some models in exchange for Kimi and DeepSeek deployments silently failing a chunk of their tool calls.
Measure your own deployment with K2VV
Don't take my numbers or Ubicloud's on faith; test your own deployment. Moonshot's K2 Vendor Verifier works against any OpenAI-compatible endpoint. The repo ships a pyproject.toml and uv.lock, and the tool-call sample set (half of the 4,000-request test set, which matches the 2,000 requests Ubicloud ran) lives at https://statics.moonshot.cn/k2vv/tool-calls.tar.gz.
The README's example for Kimi K2.5 served by vLLM or SGLang, with thinking turned off (Instant mode):
python tool_calls_eval.py samples.jsonl \
--model kimi-k2.5 \
--base-url YOUR_API_BASE_URL \
--api-key YOUR_API_KEY \
--concurrency 5 \
--extra-body '{"chat_template_kwargs": {"thinking": false}, "temperature": 0.6}'
Swap --model for whatever name your server reports. You get two numbers in summary.json:
- schema_accuracy: successful tool calls divided by responses that finished with tool_calls. Anything under 100% means your server is shipping broken calls.
- tool_call_f1: how closely your deployment matches Moonshot's API on when to call a tool. Moonshot's acceptable bar is above 73% for K2-thinking and 80% for K2-0905.
Run it once with your current flags and once with --tool-strict-level parameter. Use --incremental to rerun only the failed requests. If the schema number doesn't move, your problem is somewhere else, but at least you've ruled out the cheapest fix.
The checklist:
1. Upgrade to vLLM 0.31 or newer.
2. Add --tool-strict-level parameter.
3. Confirm VLLM_ENFORCE_STRICT_TOOL_CALLING isn't set to false.
4. Use the xgrammar backend, not guidance.
5. If you run MTP or other speculative decoding, retest after enabling strictness.
6. Run K2VV before and after, and keep the summary.json files.
So what's your number?
My guess is most teams have never measured their tool-call failure rate. They have a vague feeling the agent is "flaky" and a retry loop that hides it. What's yours in production? Have you actually measured it, and did constrained decoding help or hurt your model? If you think retries beat grammars, tell me why in the comments.
Sources
- Ubicloud: Fixing Tool-Call Accuracy in vLLM
- Hacker News discussion
- MoonshotAI K2 Vendor Verifier
- vLLM tool calling docs
- vLLM v0.31.0 release notes
- vLLM PR #45600, PR #56268, issues #58930, #44006, #55152
Mentions
Same model, 15 providers, 84.63% to 100% tool-call accuracy Two rows out of a table Moonshot publishes in their K2 Vendor Verifier repo: | Provider | schema_accuracy | successful / triggered | | --- | --- | --- | | Fireworks | 100.00% | 1703 / 1703 | | Together | 84.63% | 1602 / 1893 | Same weights. Same 4,000-request test set. Same reference to compare against (Moonshot's own endpoint decides what a correct response looks like). The only variable is who served the model. Moonshot built that harness after a run of "K2 is bad at tool calling" reports that turned out to be deployment problems. Their write-up is direct about why: the more open the weights, the more channels they get deployed through, and the less control anyone has over the result. From the outside you can't tell a model defect from an engineering deviation, and if users can't tell, the model gets blamed for both. The two metrics measure different failures tool_call_f1 scores the decision to call a tool at all. It's computed off finish_reason: true positive when the deployment and the reference both return tool_calls, false positive when only the deployment does, false negative when only the reference does. It answers one question. Does this endpoint decide to call tools at the same moments the official API does. schema_accuracy is count_successful_tool_call / count_finish_reason_tool_calls. It only counts responses that already committed to a tool call, then asks how many produced arguments that validate against the JSON schema. The second metric is the one that reaches your error handler. finish_reason says tool_calls, your dispatcher pulls the arguments out, the payload doesn't match the schema, and you get something like this in the middle of a 40-step run: jsonschema.exceptions.ValidationError: 'path' is a required property at tools.read_file(arguments={'file': 'src/api/handlers.ts'}) A retry sometimes papers over it. It costs you a round trip and a chunk of context either way, and in a loop that is already deep into a plan, the retry happens without the model knowing why. The spread, on identical weights Kimi K2 Thinking, 15 deployments, temperature 1.0: | Provider | schema_accuracy | successful / triggered | | --- | --- | --- | | MoonshotAI (official) | 100.00% | 1958 / 1958 | | Fireworks | 100.00% | 1703 / 1703 | | InfiniAI | 99.89% | 1825 / 1827 | | SiliconFlow | 98.96% | 2097 / 2119 | | GMICloud | 95.95% | 1775 / 1850 | | AtlasCloud | 95.74% | 1798 / 1878 | | SGLang | 95.52% | 1790 / 1874 | | vLLM | 87.22% | 1856 / 2128 | | Parasail | 87.14% | 1837 / 2108 | | DeepInfra | 86.91% | 1800 / 2071 | | GoogleVertex | 85.76% | 1668 / 1945 | | Together | 84.63% | 1602 / 1893 | | Chutes | 83.05% | 3037 / 3657 | Sixteen points between the top and the bottom of the same model. On the K2 0905 table the floor is lower: five deployments at 100%, then vLLM at 76.00%, SGLang at 73.13%, Volc at 72.86%, Baseten at 72.49%, Together at 71.96%. Roughly 28 points of spread on identical weights. A perfect schema score can still be a broken deployment Look at Groq on the 0905 table: 100.00% schema accuracy, and a trigger F1 of 69.52%. The schema metric only looks at calls the model already decided to make, so a deployment that fires tool calls at the wrong moments keeps a clean schema score. Nebius shows the same shape, 84.47% schema with 50.60% F1. Read the pair together or the number lies to you. High schema plus low F1 means the endpoint is confident about when to call tools and wrong about it. Per-call accuracy is the wrong unit Schema accuracy is a per-call number. Your agent doesn't make one call, it makes a session's worth, and every call has to validate. If the per-call probability is p, the chance that all N calls in a session validate is p^N. | Provider | per call | 10 calls | 25 calls | | --- | --- | --- | --- | | Fireworks | 100.00% | 100.0% | 100.0% | | SiliconFlow | 98.96% | 90.1% | 77.0% | | GMICloud | 95.95% | 66.1% | 35.6% | | vLLM | 87.22% | 25.5% | 3.3% | | Together | 84.63% | 18.9% | 1.5% | | Chutes | 83.05% | 15.6% | 1.0% | A 1.04 point gap between SiliconFlow and Fireworks looks like noise per call. Over a 25-call session it's a 23 point gap. This is the same reason a cost-per-PR number is a distribution rather than a price: the interesting behavior lives in the tail of the repeated event, not in the average of one. Two honest caveats on the table above. The denominators differ per provider (Moonshot's official run triggered 1958 tool calls, Together's 1893, Chutes' 3657), so these are not perfectly paired samples. And a schema failure in production is often a retry rather than a dead run, so p^N is a lower bound on session success, not a prediction of crash rate. The shape holds either way: small per-call differences get amplified by repetition, and the amplification is where agents live. The same provider shows up at both ends DeepInfra is 100.00% on k2-0905-preview at temperature 0.6, and 86.91% on k2-thinking at temperature 1.0. Together is 71.96% on one table and 84.63% on the other. So a provider's score is a property of a specific model version, decoding parameters, and serving stack, not a brand fact you can carry forward. KVV's first fix after the initial reports was exactly this class of thing: enforcing Temperature=1.0 and TopP=0.95 in thinking mode, and requiring that thinking content be passed back to the API. The repo also names two failure modes that are worth reading before you file a bug against a model. Tool call IDs have to follow the format functions.func_name:idx, and Moonshot's own API rewrites incoming IDs into that shape automatically. A provider that doesn't rewrite them feeds the model malformed IDs from history and gets parsing failures back. And there's no built-in schema enforcement: the model generates tokens by probability, so the README tells vendors to add guided encoding rather than trust prompting. Their recommended versions are vLLM v0.11.0 or newer and SGLang v0.5.3rc0 or newer for 0905, with v0.11.1rc6 and v0.5.5.post2 for thinking. If you route through an aggregator, you're already measuring a moving target Most teams don't pin a provider. They point at a router and take whatever answers. OpenRouter's default is price-based load balancing across providers with allow_fallbacks set to true, so the endpoint that serves your request can change between runs without any config change on your side. Their docs are explicit that a sort or order preference disables load balancing. That means your eval score, your p95 latency, and your schema accuracy are all attached to a provider you didn't choose. For an eval run, pin it: extra_body={"provider": {"only": ["fireworks"], "allow_fallbacks": False}} If you run a reviewer or an agent on your own keys, the provider is a config field you own rather than a default you inherit. Kodus's BYOK setup has that shape, keys in and a model choice per task, and it's the right shape for this problem: the provider is a variable you set per job and can hold constant while you measure something else. A protocol you can run in an afternoon 1. Pick one model version and one decoding parameter set. Write both down, because the next table you compare against won't have them. 2. Run your own tool-call test set against at least two providers, and log finish_reason plus a schema validation result per response. Do not log only pass or fail on the session, or you'll lose the per-call number. 3. Report schema_accuracy next to trigger F1. A clean schema score on its own has already been shown to hide a deployment that calls tools at the wrong times. 4. Re-run after every version bump on either side, and treat a provider change as a model change for eval purposes. The KVV table is a deployment check, not a model review. The useful takeaway isn't which vendor is on top this month, it's that identical weights produce a 16 to 28 point spread in the number your agent depends on, and that spread is something you can reproduce on your own endpoint in an afternoon. If you can't reproduce it, you don't have a number. You have a vendor's number.
vLLM Tool Calling Not Working? 1 in 12 Calls Fail by Default. Here's the Fix If your vLLM tool calling is not working with Kimi or DeepSeek, and your agent keeps emitting broken JSON, inventing arguments or looping on the same call, check your server before you blame the model. Ubicloud measured Kimi K2.6 on vLLM 0.28 and got schema-valid tool calls only 92% of the time. That is 2 failures in every 25 calls, roughly 1 in 12. With constrained decoding turned on, Ubicloud got to 100%. vLLM doesn't turn it on for you, even though Moonshot, who make the model, tell vendors to use it (they call it guided encoding). On vLLM 0.31+ the fix is one server flag, --tool-strict-level parameter, and the rest of this post is about why and how to check it worked. A lot of the "open models are bad at agents" talk I've seen comes down to a server default. What broken vLLM tool calls look like On the client it rarely says "tool call failed". Instead you get: - arguments that won't parse as JSON, or parse but miss a required field - a parameter the tool never declared, or an enum value that isn't in the enum - the agent reading back its own validation error, retrying with the same mistake and burning your token budget in a loop - a framework that quietly drops the call and lets the model answer in prose Each of these looks like a model problem, so people swap models or add three paragraphs of "ALWAYS RETURN VALID JSON" to the system prompt. Neither fixes it. 92% sounds fine until you chain calls Per-call accuracy compounds. At 92%, a 10-step agent run gets through clean 0.92^10 ≈ 43% of the time. At 20 steps it's about 19%. In the Hacker News thread, one commenter worked out that even 95% per call gives you about an 8% chance of finishing a 50-call sequential task. Someone else in the same thread said they run around 300 tool calls per prompt. 92% wasn't a fluke of Ubicloud's setup, either. Moonshot's own K2 Vendor Verifier results (test date 2025-11-15) show the vLLM reference deployment at 87.22% schema accuracy on K2-thinking and 76.00% on K2-0905, while Moonshot's API and providers like Fireworks hit 100% serving the same weights. Why vLLM leaves tool calls unconstrained by default Without constraints, the model samples tool-call tokens like any other text. Kimi K2.6's format looks like this: <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|> Nothing stops the model from writing an unknown function name, a non-integer index or JSON that ignores your schema. Constrained decoding uses XGrammar structural tags to mask the vocabulary at each step, so once the model opens a tool call it can only produce a declared tool name, then an integer index, then JSON that matches that tool's schema, then the end tag. vLLM supports this, just not on every request. PR #45600, merged in June 2026, changed tool_choice="auto" so structural tags apply only when at least one tool sets "strict": true. A missing strict counts as not strict. A reviewer on that PR objected that Claude Code's protocol doesn't include a strict field at all, which left Messages API clients with no obvious opt-in; the PR ended up adding strict to vLLM's Anthropic tool model. Plenty of OpenAI-style agent code doesn't set it either. So if you're on tool_choice="auto" (most agents are) without strict on your tools, you're sampling free-form. Ubicloud originally added constrained decoding in their own vLLM fork. After vLLM shipped the feature in v0.25 they switched to upstream, and their score dropped back to 91% because of this default change. The fix: --tool-strict-level in vLLM 0.31+ vLLM 0.31.0 added a server-side floor, --tool-strict-level (PR #56268). From the tool calling docs: | Value | What gets constrained | |---|---| | auto | Default for most parsers. Structural tags only for required/named tool choice, or auto when a tool sets strict: true | | function | Tool-call envelope (markup, function name) on every request with tools. Arguments stay free unless the tool is strict | | parameter | Everything in function, plus argument schemas for every tool, as if all were strict | If you don't set it, the parser's default applies: function for glm47, auto for everything else, Kimi included. For a Kimi deployment I'd go straight to parameter. Invented args and missing required fields are argument problems, and function doesn't touch those: vllm serve <your-kimi-k2.6-checkpoint> \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 \ --tool-strict-level parameter Two things to know before you deploy it: - It's a floor. It never loosens what a request already asked for, and it doesn't force a tool call. The grammar engages only after the model opens the tool-call wrapper itself. - VLLM_ENFORCE_STRICT_TOOL_CALLING=false turns structural tags off entirely and overrides this flag. If someone set it while chasing a bug months ago, the flag does nothing. Stuck below 0.31? Set "strict": true on every function definition you send. Under tool_choice="auto" that turns on the same structural tags without a server flag. Where constrained decoding still bites This has costs, and the HN thread argues about them at length. A grammar guarantees valid syntax but can't make the arguments correct: a well-formed call to the wrong tool with a wrong URL still passes. One commenter argued that grammar masking only gives you token-level retries rather than retrying the whole sequence, so it can skew output away from what the model would have picked, and suggested just retrying failed calls. They later allowed that retries might not win once you count cost. Ubicloud's post calls constrained decoding "a per-model task", and one of its authors said in the thread that Qwen 3.8 doesn't come with the clear recommendation Moonshot gives for Kimi, and advised checking model quality with it on and off. Measure both. It also has bugs you should know about: - #58930: JSON schemas nested inside structural tags skip XGrammar's unsupported-feature check, so a constraint like multipleOf is silently ignored. Still open at the time of writing, with a fix proposed in PR #58940. - #44006: strict tool calling plus MTP speculative decoding threw "Failed to advance FSM" errors and 500s. Closed by PR #44297 in July 2026, though a later comment on v0.28.0 still saw the FSM error, just without the 500. - #55152: the guidance backend rejects the new structural tag format with KeyError: 'triggers'. A vLLM contributor replied that the tool-call structural tags were built for xgrammar and expects all of them to fail on guidance. Ubicloud also notes that gpt-oss-20b, Qwen 3.6 and GLM 5.1 users reported issues back when constrained decoding was on by default. PR #45600 says it wanted to stop structural tags from over-constraining generation when the caller hadn't asked for strict. I think that was the wrong trade: a few loud regressions on some models in exchange for Kimi and DeepSeek deployments silently failing a chunk of their tool calls. Measure your own deployment with K2VV Don't take my numbers or Ubicloud's on faith; test your own deployment. Moonshot's K2 Vendor Verifier works against any OpenAI-compatible endpoint. The repo ships a pyproject.toml and uv.lock, and the tool-call sample set (half of the 4,000-request test set, which matches the 2,000 requests Ubicloud ran) lives at https://statics.moonshot.cn/k2vv/tool-calls.tar.gz. The README's example for Kimi K2.5 served by vLLM or SGLang, with thinking turned off (Instant mode): python tool_calls_eval.py samples.jsonl \ --model kimi-k2.5 \ --base-url YOUR_API_BASE_URL \ --api-key YOUR_API_KEY \ --concurrency 5 \ --extra-body '{"chat_template_kwargs": {"thinking": false}, "temperature": 0.6}' Swap --model for whatever name your server reports. You get two numbers in summary.json: - schema_accuracy: successful tool calls divided by responses that finished with tool_calls. Anything under 100% means your server is shipping broken calls. - tool_call_f1: how closely your deployment matches Moonshot's API on when to call a tool. Moonshot's acceptable bar is above 73% for K2-thinking and 80% for K2-0905. Run it once with your current flags and once with --tool-strict-level parameter. Use --incremental to rerun only the failed requests. If the schema number doesn't move, your problem is somewhere else, but at least you've ruled out the cheapest fix. The checklist: 1. Upgrade to vLLM 0.31 or newer. 2. Add --tool-strict-level parameter. 3. Confirm VLLM_ENFORCE_STRICT_TOOL_CALLING isn't set to false. 4. Use the xgrammar backend, not guidance. 5. If you run MTP or other speculative decoding, retest after enabling strictness. 6. Run K2VV before and after, and keep the summary.json files. So what's your number? My guess is most teams have never measured their tool-call failure rate. They have a vague feeling the agent is "flaky" and a retry loop that hides it. What's yours in production? Have you actually measured it, and did constrained decoding help or hurt your model? If you think retries beat grammars, tell me why in the comments. Sources - Ubicloud: Fixing Tool-Call Accuracy in vLLM - Hacker News discussion - MoonshotAI K2 Vendor Verifier - vLLM tool calling docs - vLLM v0.31.0 release notes - vLLM PR #45600, PR #56268, issues #58930, #44006, #55152