SWAM: Scaling World Action Models with Cross-Modal Alignment
SWAM: Scaling World Action Models with Cross-Modal Alignment
SWAM extends the World Action Model through cross-modal alignment, trained on 47,000 hours of multimodal data, blending egocentric video and UMI demonstrations to enhance control generalization, ultimately achieving a 40.00% success rate on three real-world tasks, surpassing the baseline of 23.33%.
Published on Nov 9
Authors:
,
,
,
,
,
,
,
,
,
Abstract
World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.
View arXiv page View PDF Project page GitHub 6 Add to collection
Get this paper in your agent:
hf papers read 2611.12870
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.12870 in a model README.md to link it from this page.
Datasets citing this paper 1
FreDERicia/SWAM-Pretrain
Preview • Updated 18 minutes ago • 709 • 3
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.12870 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co