Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success
Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success
The Sensorimotor Alignment (S²) metric finds that the alignment degree between vision and motor representations of embodied agents is positively correlated with task success rates, verifying the applicability of the Platonic representation hypothesis in the robotics field. In simulation and real robot tests, models with high S² (such as Vision-Language-Action and World Action Models) perform better and can explain the phenomenon where the VLA model ignores instructions and relies on visual features, serving as a low-cost offline proxy metric.
The Platonic Representation Hypothesis (Huh et al, 2024) argues that as vision and language models scale, their representation spaces converge toward a shared statistical model of reality. The idea is that images and text are different projections of the same underlying world. As models recover that world more faithfully, they begin to agree on which inputs are similar. For example, if a vision model places two images close together in the representation space, a language model tends to place their captions close together too.
In this work, we bring PRH to robotics and ask: can physical robot motion be viewed as another projection of that same world, such that more capable embodied models better align their vision and language representations with the structure of motion trajectories?
Our evidence suggests the answer is yes. To test this, we use Sensorimotor Alignment (S²), which measures the overlap between neighborhoods of a model's sensory representations and neighborhoods of end-effector trajectories. Higher alignment means episodes nearby in sensory representation space are also nearby in motor representation space.
Our first finding is that S² is positively related to downstream robot task success. We test a range of Vision-Language-Action models (VLAs) and World Action Models (WAMs) in simulation, as well as policies on real robots. Across these settings, models with higher S² tend to achieve higher task success rates. This pattern provides empirical support for extending PRH to robotics.
S² also explains, at the representation level, several phenomena that prior work has observed only through closed-loop results. For example, VLAs often ignore the language instruction and act mainly on what they see. We find that their task success correlates with the S² of their vision features, not their text features.
In practice, S² can serve as a low-cost offline proxy for identifying which models are likely to achieve higher task success before closed-loop evaluation. For instance, it can identify promising vision-language model (VLM) backbones even before costly VLA fine-tuning.
Source: Hugging Face Daily Papers · huggingface.co