DyCraft: A System for Creating Long-Term 3D Animations

Paper: DyCraft: An Agentic System for Long-Horizon 4D Scene Generation

Authors: Jeff Tan, Hang Dai, Zeqian Long, Yanzhe Lyu, R. Kenny Jones, Shangzhe Wu, Chen Geng, Jiajun Wu

Executive Summary

Recent research introduces DyCraft, a novel system designed for generating animated 3D scenes, addressing a notable limitation in current coding agents. While these agents can create high-quality 3D models, they often struggle with producing coherent long-term animations. This challenge primarily stems from their tendency to generate physically implausible outputs, such as objects that overlap unnaturally or move in ways that defy the laws of physics.

DyCraft aims to improve this process by employing a method inspired by human animators. Instead of trying to produce an animation in one go, DyCraft breaks the task into manageable steps. Initially, it takes a user’s prompt and expands it into a detailed plan before constructing a static scene. This preparatory phase helps to set clear expectations. The system then creates a 3D storyboard, which consists of key animation poses. Each of these storyboard frames represents a static scene that the coding agents can build and troubleshoot independently. This step-by-step approach allows for a more organized animation process, focusing on pairs of verified starting and ending states. As a result, the agents can better manage the local motion between these states using various motion authoring tools.

The research evaluated DyCraft's performance across 24 animation tasks, which included humanoid robots, human and animal characters, and mechanical objects. When tested against a native GPT-6 agent integrated within the Codex framework and previous agent-based models, DyCraft demonstrated a significant reduction in physical errors. It produced higher-quality animated scenes, which were also preferred by human evaluators.

The implications of this research are substantial for industries reliant on 3D animation, such as gaming, film, and virtual reality. By improving the reliability and quality of automated animations, DyCraft could streamline production workflows, reduce the need for extensive manual corrections, and ultimately enhance the creative process. The results were obtained through simulations and benchmarks, indicating a controlled testing environment rather than real-world applications. However, its potential to transform animation practices makes DyCraft a noteworthy development in the field of computer-generated imagery.

Academic Abstract

While frontier coding agents can generate high-quality 3D scenes and articulated assets, they struggle to create coherent long-horizon animations. When equipped with general-purpose harnesses, they frequently produce physically implausible outputs, generating objects that interpenetrate, lack physical support, or move unnaturally. Such bare agents struggle to repair these errors because they default to scripting the full animation in one pass, making it difficult to verify their work and disentangle complex spatiotemporal constraints. Drawing on the practice of human animators, we introduce DyCraft, a system for animated 3D scene generation that breaks animation into steps that coding agents can more reliably solve. DyCraft first expands the user's prompt into a detailed plan and builds a static scene. It then creates a 3D storyboard of key animation poses, where each frame is a static 3D scene that the agent can build and debug in isolation. Storyboarding decomposes the problem into pairs of verified start and end states, allowing the agent to author local motion between them with a collection of motion authoring tools. We evaluate DyCraft on 24 long-horizon, multi-shot animation tasks covering humanoid robots, human and animal characters, and physical mechanisms. Compared with a native GPT-6 agent in Codex and prior agentic baselines, DyCraft substantially reduces physical violations and generates higher-quality animated 3D scenes that are more often preferred by human raters.