跳到正文
实时
Hugging Face Daily Papers·· 21 小时前

AURA 2.0 流媒体视频大语言模型发布:可主动对话、调用工具并实现交互式视觉定位

Beyond Answering: Learning to Speak First and Act in Streaming Video

要点速览

AURA 2.0 是一款流媒体视频大语言模型,可主动发起对话、调用工具并实现交互式视觉定位,具备长时程记忆能力。其核心升级包括模型主动对话(MIC)、流媒体感知工具调用(SATC)和交互式视觉定位(IVG),通过三阶段训练方案提升流媒体交互效果。在公开流媒体基准测试中,AURA 2.0 在开源模型中表现领先,新交互能力得到针对性评估验证。

正文

Published on Nov 15

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

AURA 2.0 is a streaming video large language model that can initiate conversations, call tools, and perform interactive visual grounding, with long-horizon memory and a three-stage training scheme for improved streaming interaction.

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.17048

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

aurateam/AURA_2

9B •

Updated about 4 hours ago

•

5

•

13

aurateam/AURA_2_BF8

9B •

Updated about 4 hours ago

•

4

•

9

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.17048 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

来源: Hugging Face Daily Papers · huggingface.co