Sensorimotor Alignment：一种基于感知运动对齐的具身智能体成功代理指标
Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success
Sensorimotor Alignment (S²) 指标发现具身智能体的视觉与运动表征对齐程度与其任务成功率正相关，验证了柏拉图表征假说在机器人领域的适用性。在模拟与真实机器人测试中，高 S² 的模型（如 Vision-Language-Action 和 World Action Models）表现更优，且能解释 VLA 模型忽略指令而依赖视觉特征的现象，可作为低成本离线代理指标。
The Platonic Representation Hypothesis (Huh et al, 2024) argues that as vision and language models scale, their representation spaces converge toward a shared statistical model of reality. The idea is that images and text are different projections of the same underlying world. As models recover that world more faithfully, they begin to agree on which inputs are similar. For example, if a vision model places two images close together in the representation space, a language model tends to place their captions close together too.
In this work, we bring PRH to robotics and ask: can physical robot motion be viewed as another projection of that same world, such that more capable embodied models better align their vision and language representations with the structure of motion trajectories?
Our evidence suggests the answer is yes. To test this, we use Sensorimotor Alignment (S²), which measures the overlap between neighborhoods of a model's sensory representations and neighborhoods of end-effector trajectories. Higher alignment means episodes nearby in sensory representation space are also nearby in motor representation space.
Our first finding is that S² is positively related to downstream robot task success. We test a range of Vision-Language-Action models (VLAs) and World Action Models (WAMs) in simulation, as well as policies on real robots. Across these settings, models with higher S² tend to achieve higher task success rates. This pattern provides empirical support for extending PRH to robotics.
S² also explains, at the representation level, several phenomena that prior work has observed only through closed-loop results. For example, VLAs often ignore the language instruction and act mainly on what they see. We find that their task success correlates with the S² of their vision features, not their text features.
In practice, S² can serve as a low-cost offline proxy for identifying which models are likely to achieve higher task success before closed-loop evaluation. For instance, it can identify promising vision-language model (VLM) backbones even before costly VLA fine-tuning.
来源: Hugging Face Daily Papers · huggingface.co