HERO: Optimizing Prompts for Better Program Generation Performance

Paper: HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation

Authors: Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw

Executive Summary

Recent research focuses on enhancing the performance of large language models (LLMs) through a method called harness optimization. This approach is designed to improve the way prompts—essentially, the questions or commands given to the model—are structured and refined. However, the research indicates that current techniques need further development, particularly for tasks involving program generation, such as converting natural language to SQL or Python code.

A significant issue with existing methods is their reliance on LLM reflectors. These reflectors can suggest updates to prompts based on observed performance. While they can address specific failures in generating code, they may inadvertently disrupt previous successful outputs. This inconsistency is a key challenge in ensuring reliable performance across different tasks.

To tackle this problem, the researchers developed a novel approach called HERO (Harnessing Evidence for Refinement and Optimization). This method combines two phases: offline learning and test-time refinement. In the initial phase, HERO learns from executing generated code, identifying both successful and unsuccessful outputs. This allows the model to validate potential prompt revisions before they are implemented. During the test phase, the generated code is treated as a unique harness, which is then refined based on real-time feedback from code execution.

The effectiveness of HERO was evaluated using four benchmarks: SQL, Cypher, and Python. The results from these tests indicate that HERO significantly outperforms previous methods. Specifically, it achieved an improvement of 7.6 percentage points in pass rates compared to the original prompts. Moreover, the test-time refinement process added an additional 2.0 percentage points in performance and contributed to improved execution efficiency.

The implications of this research are notable for businesses and developers working with LLMs in programming tasks. By adopting HERO, companies could enhance their models' accuracy and reliability when generating code. This could lead to more efficient workflows in software development and data analysis, where translating natural language into code is crucial. The findings suggest that with further implementation and testing, HERO could serve as a valuable tool for improving program generation capabilities in various applications.

Academic Abstract

Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \textsc{HERO} (\textbf{H}arnessing \textbf{E}vidence for \textbf{R}efinement and \textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \method{} achieves the best offline performance, improving pass@1 by (7.6) points over the initial prompt, while test-time refinement adds another (2.0) points and improves execution efficiency.