Fixing Tool-Call Accuracy in vLLM

Tool calling lets an AI agent invoke external functions, such as fetching a webpage, by providing the model with tool descriptions and schemas that specify valid argument names, types, and required fields. The inference server converts the model’s tool-call output into a structured request for the agent to execute. If the model produces arguments that violate the schema, the call fails; error messages then clutter the conversation, degrade later responses, and may force costly retries. This is a persistent risk because models generate tokens probabilistically rather than checking their output like a compiler, and may confuse current tools or fields with patterns learned from other tool-calling setups. Moonshot AI’s K2VV ToolCall benchmark tests 2,000 requests, measuring both whether a model chooses to call a tool similarly to Moonshot’s service and whether its calls conform to the supplied schema.

The authors report that Kimi K2.6 served with vLLM 0.28 produced schema-valid calls only 92% of the time - about two failures in every 25 attempts. They explain that constrained decoding addresses this by excluding token choices that would lead to an invalid tool call, rather than relying on the model to follow instructions perfectly. vLLM added this capability and exposed it through a server-side flag in version 0.31. With the feature enabled, newer Kimi and DeepSeek models can reach 100% schema accuracy. The central point is that reliable tool use depends not only on a model’s reasoning ability but also on inference-time safeguards that ensure generated calls are executable.