SciHorizon-eLab: Streamlining Scientific Experimentation with Embodied Agents

Paper: SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

Authors: Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu, Zirui Liu, Hongting Niu, Yuanchun Zhou, Hengshu Zhu

Executive Summary

Recent research introduces SciHorizon-eLab, a new tool designed to enhance the automation of scientific experimentation through the use of embodied agents. These agents can perform tasks similar to human scientists but have been limited by the challenge of evaluating their performance reliably. The traditional methods for creating simulated laboratory environments involve extensive manual task development, which complicates the integration of a wide range of scientific protocols into tasks that can be consistently executed and verified.

SciHorizon-eLab aims to streamline this process by treating the transition from scientific protocols to actionable tasks as a compilation issue. It takes natural-language descriptions of experiments and systematically converts them into tasks that can be executed by embodied agents. The system uses a multi-step approach that includes semantic grounding, where the language is interpreted into meaningful actions, followed by the synthesis of executable tasks, and finally, simulation-based certification to ensure reliability.

One of the key outputs of SciHorizon-eLab is a benchmark named \BenchName, which includes 300 certified tasks covering various laboratory operations. This benchmark allows for hardware-in-the-loop (HIL) task execution, enabling real-world applications of the compiled tasks. It also supports the generation of expert demonstrations and provides a structured evaluation framework that includes step-by-step success specifications.

The research highlights that even with this advanced system, the top-performing policy achieved an average success rate of only 49.7% in executing these tasks. This suggests that while the tool enables significant advancements in task automation, there are still notable challenges, particularly in how well embodied agents coordinate with human operators. This finding is critical, as it indicates room for improvement in developing more effective cooperation between human scientists and robotic agents.

The authors have made their code, data, and evaluation toolkit publicly available, which may facilitate further research and development in this field. By providing a structured approach to converting scientific protocols into executable tasks, SciHorizon-eLab could potentially enable more efficient and reliable automation in laboratories, contributing to improved productivity in scientific research.

Academic Abstract

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.