跳到正文
实时
Hugging Face Daily Papers·· 22 小时前65

TCP4D:一个用于机器人操作的统一工具中心点跟踪与生成模型

TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

要点速览

TCP4D模型直接从单目视频和初始2D TCP查询中恢复SE(3)框架下的TCP位置和抓取器状态,同时生成未来TCP轨迹。其跟踪器在DROID和RoboLab上实现7.60和16.07毫米的绝对轨迹误差,仅用十个演示即可适应新形态。

正文

Published on Nov 17

Authors:

,

,

,

,

,

,

Abstract

Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.

View arXiv page View PDF Project page GitHub 7 Add to collection

来源: Hugging Face Daily Papers · huggingface.co