CGAP: Streamlining Natural Language Requests for LLM Agents

Paper: Capability-Grounded Agentic Planning

Authors: Shikha Shrestha, William Yeoh, Ning Zhang, Yevgeniy Vorobeychik

Executive Summary

Recent research introduces a new approach called Capability-Grounded Agentic Planning (CGAP), which aims to enhance how large language model (LLM) agents handle natural-language requests. Traditional LLM agents often tackle requests through a trial-and-error method, which can lead to inefficiencies and uncertainty in generating executable plans. CGAP seeks to improve this process by providing a systematic way to determine the necessary steps and tools for fulfilling a request.

The core innovation of CGAP lies in its structured planning methodology. Instead of randomly selecting calls to tools or data sources, CGAP first creates a detailed model of the available tools and their capabilities. This model is built using a formal planning representation known as PDDL (Planning Domain Definition Language). The LLM then translates a user’s request into specific goals based on this model, allowing for a more organized and predictable planning process.

One key feature of CGAP is its ability to handle uncertainty around tool outputs. The model remains abstract, focusing on which inputs are needed rather than the values those inputs might have. This means that it can adapt to various scenarios without being constrained by predetermined outcomes. During execution, the LLM adds specific calls that were not included in the initial plan, ensuring that the response remains relevant to the user's request. Additionally, a verification step is included where an LLM checks that the responses align with the original request before finalizing the output.

The effectiveness of CGAP has been demonstrated through benchmarks using multiple datasets, including ToolSandbox, ComplexFuncBench, BFCL multi-turn, and τ²-bench retail. In these tests, CGAP has shown significant performance improvements, outperforming traditional tool-calling agents by up to 28.9 points. This improvement suggests that businesses could benefit from implementing CGAP to enhance the efficiency and reliability of their LLM applications.

The implications of this research are particularly relevant for sectors that rely on automated customer interactions, data processing, and decision-making systems. By adopting a more structured approach to request handling, businesses may achieve quicker response times and more accurate outputs, ultimately leading to better customer satisfaction and operational efficiency.

While the research has been conducted in controlled environments, the principles of CGAP could be applied in real-world scenarios across various industries, including retail, customer service, and data analytics. As organizations increasingly leverage AI technologies, the advancements presented by CGAP offer a promising direction for improving the capabilities of LLM agents to meet complex user needs effectively.

Academic Abstract

LLM agents serve natural-language requests through tools, data sources, and other agents, and even simple requests often need several interdependent calls. Typically, the LLM picks these calls by trial and error, with no guarantee of an executable plan. We present Capability-Grounded Agentic Planning (CGAP), which instead reasons about what a request requires and what tools provide. CGAP compiles the tool catalog deterministically into a PDDL model, uses an LLM to turn the request into goals, and solves the model with a classical planner. Since tool outputs are unknown in advance, the model is abstract, tracking which inputs are available, not their values. During execution, an LLM adds entity-specific calls that abstract plans omit, and deterministic checks reject any that break the plan. Every argument is drawn from a cited source, such as a conversation span or an earlier tool response, and each requested result counts as delivered only after an LLM verifier checks it against the request. With three LLMs on ToolSandbox, ComplexFuncBench, BFCL multi-turn, and $\tau^2$-bench retail, CGAP outperforms a standard tool-calling agent by up to 28.9 points.