NeoWorld-3: Interactive 3D Environments for Real-World Applications

Paper: NeoWorld-3: From Pixels to Code-Defined Physical Worlds

Authors: Eachen Soong, Zijian Niu, Haoran Xu, Yihong Dong, Yunbo Wang, Xiaokang Yang

Executive Summary

NeoWorld-3 is a new framework designed to create interactive 3D environments that go beyond simple visual representations. Current 3D generation tools often lack the ability to provide physical interactivity, which is essential for applications in fields such as robotics, gaming, and virtual training. NeoWorld-3 addresses this gap by allowing users to construct 3D worlds that can be manipulated in a realistic manner.

At the heart of NeoWorld-3 is the NeoSDK, a computational engine that facilitates the development of low-level geometries and the composition of parts within a 3D scene. This engine allows for fine-tuning of parameters and validation of physical interactions, ensuring that the generated environments behave as they would in the real world. Additionally, the NeoMCP component provides a user-friendly interface through structured application programming interfaces (APIs). This means that vision-language agents can visually inspect and edit the code used to create these 3D worlds, enabling iterative refinement.

The framework is designed with a hierarchical architecture, allowing it to scale from simple objects to more complex scenes without losing performance or fidelity. This scalability is crucial for various applications that require detailed and dynamic environments.

Benchmark evaluations indicate that NeoWorld-3 achieves high levels of geometric accuracy and is physically ready for interaction. Notably, the framework has demonstrated its capabilities through successful execution of manipulation tasks on a physical dual-arm robot. These tasks were planned within the reconstructed 3D environments, showcasing the framework's ability to function in a Real-Sim-Real loop, where simulations can directly inform and enhance real-world operations.

This research could have significant implications for industries that rely on accurate and interactive 3D models. For example, in robotics, it may improve training simulations, allowing robots to learn from virtual environments before operating in the real world. In gaming, developers could create more engaging and responsive experiences by utilizing these interactive worlds. Overall, NeoWorld-3 represents a promising advancement in the creation of realistic and manipulable 3D environments, with potential applications across various sectors.

Academic Abstract

Simulation-ready world reconstruction demands physical interactivity beyond visual realism, which existing black-box 3D generation fails to deliver. We present NeoWorld-3, a coding-agent framework that reconstructs interactive 3D worlds from visual observations via tunable, executable programs. At its core, NeoSDK provides the underlying computational engine for low-level geometries, part composition, differentiable parameter refinement, and physics validation. At the same time, NeoMCP exposes these capabilities as structured, tool-callable APIs for vision-language agents to visually inspect, edit code, and iteratively refine programs. Through a hierarchical coordinator-worker architecture, the framework scales seamlessly from standalone articulated objects to complex compositional scenes. Evaluations across object-level and scene-level reconstruction benchmarks demonstrate that NeoWorld-3 achieves state-of-the-art geometric fidelity and physical readiness. Crucially, manipulation trajectories planned within our reconstructed twins achieve successful zero-shot execution on a physical dual-arm robot, completing the Real-Sim-Real loop.