VidUEU-Agent：一个用于多模态理解、编辑和统一任务的数据管理智能体
VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
VidUEU-Agent框架通过从原始视频自动构建指令数据，解决多模态模型开发中高质量多模态指令数据的稀缺和高标注成本问题。该框架采用四阶段数据构建流程，结合视觉语言模型和视觉工具进行证据支持的数据合成，并利用下游训练反馈进行自主迭代。
🚀 We are excited to share our latest work: “VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks”!
The scarcity and high annotation cost of high-quality multimodal instruction data remain major bottlenecks for multimodal model development. Raw videos contain rich visual knowledge and temporal information, but transforming them into accurate, verifiable, and task-oriented instruction data remains challenging.
To address this, we introduce VidUEU-Agent, an agentic data-curation framework that automatically constructs instruction data from raw videos for multimodal understanding, image editing, and unified understanding-and-generation tasks.
🔑 Key Highlights
🎬 Four-Stage Data Construction Pipeline:
VidUEU-Agent performs global perception to analyze video structure and filter low-quality shots, mines informative keyframes and multimodal signals, constructs structured metadata, and routes each metadata entry to suitable task types for instruction synthesis. Agent planning, tool use, and multi-stage quality control support the entire process.
🔍 Evidence-Grounded Data Synthesis:
The framework combines vision-language models with visual tools for quality assessment, semantic analysis, and information verification. During instruction synthesis, the model cites the evidence signals it uses, improving data traceability and factual consistency.
🔄 Autonomous Iteration with Training Feedback:
VidUEU-Agent uses downstream training and evaluation results to identify weaknesses and refine data distributions across capabilities, domains, and task types. Across four iterations, each using 30K samples to fine-tune BAGEL from the same initial checkpoint, performance improves overall on the three feedback benchmarks:
📦 VidUEU-100K Dataset:
Using this pipeline, we construct VidUEU-100K from 8,268 diverse videos, containing nearly 20,000 dense metadata annotations and 100,000 instruction samples: 40K understanding, 30K editing, and 30K unified samples, spanning nine task paradigms and five scenario categories.
Our work demonstrates how raw videos can be transformed into diverse, evidence-grounded instruction data through agent planning, tool augmentation, and training-feedback-driven refinement. We hope VidUEU-Agent and VidUEU-100K provide a scalable and reproducible foundation for developing unified multimodal models.
📄 Paper: https://arxiv.org/abs/2611.04418
💻 Code: https://github.com/LRuca/VidUEU-Agent
🤗 Dataset: https://huggingface.co/datasets/Lruca/VidUEU100k
We welcome feedback and discussions!
来源: Hugging Face Daily Papers · huggingface.co