ACTS: Robotic Skills for Contact-Rich Manipulation Tasks

Paper: ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Authors: Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson

Executive Summary

The research introduces a new approach for developing robotic manipulation skills that involve contact-rich interactions, which are critical for tasks like handling objects or performing delicate operations. Traditional methods of training robots require extensive data collection through teleoperated robot interactions, a process that can be slow and limited by the robot's ability to sense forces and the design of its gripper. This study explores an alternative: using data from human interactions to train robots.

The authors present a system called ACTS, which stands for Action-Conditioned Tactile Sensing. This system leverages information gathered from human operators using specialized sensors, namely handheld GelSight devices that provide tactile feedback. By recording various types of data—including video footage from multiple angles, tactile images, and detailed motion data—researchers compiled a dataset called React. This dataset consists of 4.9 hours of interactions involving three different objects.

ACTS is designed to predict future scenarios based on the tactile and visual data collected from human actions. Specifically, it improves the robot's ability to anticipate how a scene will change in response to contact, as well as how tactile impressions evolve over time. The research demonstrates that ACTS outperforms existing visuo-tactile models—ones developed from scratch using the same React dataset—by showing better predictive accuracy for scene views.

In practical terms, the results indicate that robots trained with this human-derived data could learn to handle objects more effectively by understanding the dynamics of contact. Specifically, the transfer of contact dynamics from human interaction to robot execution appears to be more reliable than transferring visual aspects. This suggests that robots could potentially better predict how to manipulate objects based on tactile feedback rather than relying solely on visual input.

However, the study also notes some limitations. While the predictions related to tactile impressions were generally accurate, the visual predictions varied in reliability. For instance, while the robot could better anticipate tactile changes, its ability to accurately replicate visual features was inconsistent across different objects. This indicates that while the tactile learning process is promising, the visual aspect may still require further refinement.

This research demonstrates a shift towards using human interaction data to enhance robotic manipulation capabilities, which could lead to more effective robotic systems in various applications, such as manufacturing, healthcare, and personal assistance. The findings indicate that with further development, robots could become more adept at performing tasks that require nuanced handling of objects, ultimately improving their utility in real-world scenarios.

Academic Abstract

Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.