Evaluating LLMs for Insights from SEC Filings

Paper: From SEC Filings to Financial Signals: Evaluating LLMs on Stock Market Reactions and Analyst Forecasts

Authors: Shengkun Wang, Taoran Ji, Linhan Wang, Sattar Mansi, Chang-Tien Lu, Yinlin Chen

Executive Summary

This research evaluates how well large language models (LLMs) can interpret and analyze the Management Discussion and Analysis (MD&A) sections of Form 10-Q filings, which are regulatory documents that provide insights into a company's financial performance and outlook. The study focuses on the effectiveness of these models in generating insights that align with analyst expectations and market outcomes.

The research examines outputs from 13 different LLMs, both general-purpose and finance-specialized, across 45,964 Form 10-Q filing events spanning from 2001 to 2024. The goal is to assess how these models perform in tasks such as ranking returns based on MD&A text and forecasting earnings per share (EPS) for the following quarter. The evaluation involves linking MD&A content to financial metrics, analyst predictions, and actual market performance.

Key findings indicate that enhancing LLM outputs with cross-sectional fundamental ranks improves their ability to predict returns. Specifically, incorporating these ranks increases the mean effective-date rank correlation by 0.0089 for returns on the effective event date and by 0.0098 for returns on the following day. In contrast, adding analyst-summary context does not yield a similar improvement.

The study also notes performance differences between the types of models. Finance-specialized models performed better in return ranking on the effective event date but did not show a statistically significant advantage for the following day. In comparison, general-purpose Qwen models delivered the best individual return-ranking results.

When LLMs were tasked with forecasting EPS under a full-information view, their average performance landed in the 43.4th percentile of analyst predictions, indicating that they did not consistently outperform individual analysts. However, when forecasts were adjusted using consensus anchoring — a method that incorporates general analyst expectations — the share of forecasts that fell within an acceptable range increased from 41.1% to 59.5%. This adjustment, while improving the accuracy of the forecasts, also led to increased reporting errors in EPS figures.

This research matters to decision-makers as it highlights the potential of LLMs to enhance financial analysis and reporting processes. It indicates that while these models can provide valuable insights, their effectiveness may depend on the context and the additional information incorporated into their outputs. The findings suggest that companies and analysts could consider leveraging these models for improved forecasting and analysis, but they should remain cautious about the limitations observed in real-world applications.

Academic Abstract

Form 10-Q MD&A sections combine long-form management narrative with accounting and forward-looking information. Benchmarks for financial filings commonly evaluate question answering or numerical reasoning, leaving unclear whether outputs from large language models (LLMs) align with market and analyst outcomes. We evaluate prompted, zero-shot outputs from 13 general-purpose and finance-specialized LLMs on 45,964 Form 10-Q filing events from 2001 to 2024, linking MD&A text to cross-sectional fundamental ranks, pre-filing analyst forecasts, and realized abnormal returns. The evaluation covers return ranking under five controlled information views and next-quarter EPS forecasting under direct and consensus-anchored interfaces. Relative to disclosure alone, adding cross-sectional fundamental ranks raises mean effective-date rank correlation (RankIC) by 0.0089 for effective-event-date returns and 0.0098 for following-day returns, whereas analyst-summary context provides no comparable average gain. Across the four disclosure-containing views, finance-specialized models show an effective-event-date group advantage but no statistically reliable following-day advantage, while general-purpose Qwen models produce the strongest individual return-ranking results. Under the full-information view, direct LLM forecasts average the 43.4th percentile of the analyst distribution and do not consistently beat individual analysts. Consensus anchoring raises the mean inside-range share from 41.1% to 59.5% but increases reported-EPS error.