Beyond Answering: Learning to Speak First and Act in Streaming Video
Beyond Answering: Learning to Speak First and Act in Streaming Video
AURA 2.0 is a streaming video large language model that can proactively initiate conversations, call tools, and achieve interactive visual localization, with long-term memory capabilities. Its core upgrades include proactive model conversation (MIC), streaming-aware tool calling (SATC), and interactive visual grounding (IVG), which enhance streaming interaction through a three-stage training program. In public streaming benchmark tests, AURA 2.0 leads among open-source models, with its new interactive capabilities evaluated and validated.
Published on Nov 15
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
AURA 2.0 is a streaming video large language model that can initiate conversations, call tools, and perform interactive visual grounding, with long-horizon memory and a three-stage training scheme for improved streaming interaction.
Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.17048
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 2
9B • Updated about 4 hours ago • 5 • 13aurateam/AURA_2
9B • Updated about 4 hours ago • 4 • 9aurateam/AURA_2_BF8
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.17048 in a dataset README.md to link it from this page.
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
Source: Hugging Face Daily Papers · huggingface.co