FINCOSMOS: A New Framework for Evaluating Financial Agents

Paper: FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

Authors: Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou

Executive Summary

The research introduces FINCOSMOS, a new evaluation framework designed to assess the performance of language agents in the financial sector, where trust is paramount. Traditional benchmarks for evaluating these agents often simplify the task by fixing the data, the timeline, or the specific financial scenario being analyzed. This can lead to misleading results, as agents may operate under conditions that do not reflect real-world complexities. FINCOSMOS aims to address this issue by creating an open-world environment that better mirrors actual financial decision-making, where users must rely on agents to gather real-time data from the web without prior guidance.

In this framework, a simulated user is assigned a hidden strategy formulated from a structured set of rules. The agent, without any initial data, must independently collect information from the web, develop a backtest within a set timeframe, and ultimately present results that can be verified for accuracy. This process is rigorous: agents must pass multiple checks to ensure that their output aligns with the hidden strategy specified by the user. The evaluation does not depend on a predetermined correct answer; instead, it focuses on whether the agent can meet the user's expectations and adhere to strict correctness criteria.

The research findings indicate that across four different coding agents tested in 150 sessions, none successfully completed a task that met the necessary standards. Each session highlighted significant shortcomings, with agents failing to raise a substantial portion of the details they were expected to address. Human experts reviewed the failures and attributed 88.33% of these shortcomings to the agents' limitations. This suggests that current language agents may struggle to navigate the complexities involved in financial tasks, particularly when required to operate autonomously and in real-time.

However, the study also presents a silver lining. By utilizing the feedback generated through the FINCOSMOS evaluations, an agent equipped with persistent memory was able to learn from its failures and successfully complete all ten assigned tasks. This implies that while current agents face challenges, there is potential for improvement. With the right training and feedback mechanisms, these systems may evolve to better meet the demands of financial analysis and decision-making.

The implications of this research are significant for businesses that rely on automated systems for financial insights. It underscores the necessity for robust evaluation frameworks that reflect real-world conditions, which could lead to the development of more reliable and trustworthy financial agents. As companies consider adopting AI-driven tools for financial planning and analysis, understanding the limitations revealed by studies like this one is crucial. Investing in solutions that can learn and adapt will likely be essential for organizations aiming to enhance their analytical capabilities in the dynamic financial landscape.

Academic Abstract

Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64% of the details they were not told. Human experts attribute 88.33% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.