Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent
Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent
🎮 Can AI truly learn to play complex 3D games like humans—just by watching the screen and using a keyboard and mouse?
We’re excited to introduce Gamer, a generalist Vision-Language-Action foundation model for long-horizon, real-time game interaction, developed by the miHoYo Honkai AI R&D team!
Unlike game-specific agents, Gamer interacts with the world through the same interface humans use: screenshots, keyboard, and mouse. No privileged game states or game-specific APIs!
To make this possible, we built a complete training and inference pipeline, from large-scale pre-training to real-time reinforcement learning.
🔥 1. Scaling Game Intelligence. We leverage 10K+ hours of gameplay trajectories for continual pre-training (CPT) and systematically study scaling laws across model sizes and data mixtures, exploring how generalizable game-playing abilities emerge through scaling.
🧠 2. Teaching Agents to Think and Remember. We build a multi-level, high-quality automated annotation pipeline for gameplay data, producing ~1.2M reasoning annotations and ~223K memory annotations. Our SFT framework teaches agents when to reason, what to remember, and how to execute long-horizon tasks.
⚡ 3. Making Real-Time Interaction Truly Real-Time. We develop Streaming-vLLM, a high-concurrency inference engine optimized for continuous game interaction. Through streaming KV-cache reuse, speculative decoding, and system-level optimizations, we achieve over 20× lower inference latency in our benchmark setup, enabling real-time keyboard and mouse control.
🚀 4. Scaling RL to Hour-Long Interactions. We build large-scale RL infrastructure supporting hour-long, real-time interaction trajectories in complex game environments. Our Temporal-Aware GRPO (TA-GRPO) enables learning from long-horizon environment feedback, helping agents continuously improve their planning and execution abilities.
🎯 5. Beyond Benchmarks: Can AI Actually Play Through Games? We conduct extensive automated testing in large-scale 3D games. Across multiple main storylines in Honkai: Star Rail, Gamer successfully handles over 95% of the challenges! We also evaluate diverse side quests, navigation tasks, and challenging puzzles, demonstrating strong performance across different types of interactions.
💡 One finding we find especially exciting: Faster inference doesn't just make agents respond faster. It also makes online RL training more effective! Real-time interaction plays a critical role in both learning and deployment.
🌍 Why does this matter?
We believe games are more than entertainment or static benchmarks. They are rich, dynamic environments where agents must continuously perceive, reason, remember, plan, and act.
Gamer represents a step toward building general-purpose interactive agents, with exciting potential for automated game testing, embodied intelligence, and long-horizon autonomous systems.
Our goal isn't just to teach AI how to play games. It's to explore how AI can learn to understand, interact with, and eventually master complex virtual worlds.
🎬 Project Page & Gameplay Demos: https://mihoyo-gamer.github.io/
📄 Paper: https://arxiv.org/abs/2611.18397
💬 We'd love to hear your thoughts, questions, and feedback from the community!
If you find this direction interesting, we'd really appreciate your support on Hugging Face Daily Papers! 🚀
Source: Hugging Face Daily Papers · huggingface.co