SchemaFill: Streamlining Tool Calls for Large Language Models

Paper: SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

Authors: Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye

Executive Summary

Recent research has focused on improving how large language models (LLMs) interact with external systems through a method called SchemaFill. This method enhances the efficiency of tool calling, which is essential for applications where LLMs generate structured responses based on user requests.

In many scenarios, an LLM needs to make multiple tool calls, each requiring several arguments. Traditional methods, known as autoregressive decoding, generate these calls one piece at a time, leading to significant delays, especially when multiple arguments or calls are involved. This incremental approach can be slow and inefficient, potentially affecting the user experience negatively.

SchemaFill addresses this issue by introducing a framework for what is called "slot-parallel speculative decoding." This technique allows the model to generate potential argument values for multiple calls at the same time rather than sequentially. This parallel generation can significantly reduce latency. However, because later argument values might depend on earlier ones, independently generated values could diverge from the intended output. SchemaFill cleverly mitigates this by generating candidates for future slots while verifying their correctness against the actual sequence of calls and arguments.

The verification process is crucial. It ensures that only the correct values are used in the final output. If there are discrepancies between generated candidates and what the model expects, the model can provide corrections before finalizing the response. This combination of parallel generation and targeted verification leads to a more streamlined and efficient process for tool calling.

The effectiveness of SchemaFill has been demonstrated through benchmarks on two specific datasets, Glaive and BFCL. Results indicate that this new approach can achieve a throughput improvement of up to 4.05 times compared to traditional autoregressive decoding methods. This significant gain in speed could lead to faster responses in applications where LLMs are employed, such as customer service automation, data retrieval, and interactive applications.

The implications of this research are noteworthy. By enhancing how LLMs handle tool calls, businesses may see improved operational efficiency and user satisfaction. The ability to process multiple requests concurrently could open the door to more complex and responsive applications, making LLMs more effective in real-world scenarios.

For enterprises looking to leverage AI more effectively, adopting techniques like SchemaFill could lead to better performance in applications that rely on structured data interactions. The code for implementing this framework is publicly available, providing an opportunity for organizations to experiment with and integrate this advanced approach into their systems.

Academic Abstract

LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model's output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.05$\times$ improvement in end-to-end throughput over autoregressive decoding. Code is available at https://github.com/Czzzk/SchemaFill.