GTAGym: Evaluating AI Performance in Complex Gaming Environments

Paper: GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

Authors: Jiajun Chen, Shicheng Fang, Jiahao Lu, Yuxin Wang, Xipeng Qiu

Executive Summary

This research presents GTAGym, a new evaluation and reinforcement learning (RL) framework designed for testing agent performance in complex tasks within the popular video game Grand Theft Auto V (GTA V). The study highlights the challenges of measuring agent capabilities in AAA games where task outcomes are often difficult to assess automatically. Traditional methods rely heavily on human annotation, which can be inefficient and inconsistent.

GTAGym addresses this issue by introducing an evaluation benchmark called GTAVBench, which consists of 103 tasks derived from the main story missions of GTA V. These tasks are grouped into nine families, and the framework also includes 47 additional tasks focused on image-based shooting and sniping. The design allows for an online testing environment that supports parallel evaluations using screenshots and keyboard-mouse inputs, with automatic checks of the game state to confirm task outcomes.

The researchers applied GTAGym to evaluate six advanced language models (LLMs) using two different setups. One setup presented textual instructions for the tasks, while the other, referred to as GTAVBench-AGI, provided instructions only in the first screenshot, utilizing a system that compresses interaction history for efficiency. The highest performing model achieved an accuracy of 70.9% in the GTAVBench-AGI scenario. Despite this success, certain tasks, such as aerial navigation, evasion, and interception, proved to be particularly challenging for the models.

GTAGym also includes a scalable generator that allows users to create customized task parameters within specified ranges and templates. This feature enables the generation of both 3D and image-based environments for RL, complete with automatic signals for rewards and task completion. When applied to on-foot navigation tasks, a model called Qwen3.8-27B demonstrated improved performance, with accuracy increasing from 15% to 23% on native GTA V tasks as well as transferring skills to other task categories.

The implications of this research are significant for industries interested in training and testing AI agents in intricate environments. GTAGym serves as a robust tool for developing AI systems capable of navigating complex scenarios, potentially benefiting sectors such as gaming, robotics, and autonomous vehicles. By providing a standardized framework for evaluation, it could streamline the process of refining AI capabilities in dynamic settings.

Academic Abstract

AAA games offer a rich testbed for evaluating agent perception, reasoning, and control. However, evaluating whether agents can complete tasks in AAA games is challenging and often human-annotated because task outcomes are often difficult to verify automatically, let alone applying it for Reinforcement Learning (RL). We introduce GTAGym, an environment and framework for evaluation and RL on GTA V tasks. Its evaluation benchmark, GTAVBench, derived from GTA V main-story missions, contains 103 online tasks reconstructed across nine task families, alongside 47 image-based shooting and sniping tasks using GTA V imagery. A web server provides access to the online environment and evaluation interface, supporting parallel testing through screenshots and keyboard-mouse actions with automatic engine-state outcome checks. We evaluate six frontier LLMs on GTAVBench with textual task instructions and on its GTAVBench-AGI branch, where instructions appear only in the first screenshot and a harness automatically compresses interaction history. The best configuration achieves 70.9% overall accuracy in GTAVBench-AGI, yet aerial navigation, evasion and interception remain difficult. GTAGym also provides a scalable generator that samples task parameters within user-specified ranges and option sets under fixed templates, creating batches of 3D and image-based shooting environments for RL with automatic reward and termination signals. LoRA RL of Qwen3.8-27B on generated on-foot navigation tasks improves the in-domain performance and transfers to out-of-domain task categories, and native GTA V tasks from 15% to 23%.