VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
The VidUEU-Agent framework solves the problem of scarce high-quality multimodal instruction data and high annotation costs in multimodal model development by automatically constructing instruction data from raw videos. The framework adopts a four-stage data construction process, combining visual language models and visual tools for evidence-based data synthesis, and utilizes downstream training feedback for autonomous iteration.
🚀 We are excited to share our latest work: “VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks”!
The scarcity and high annotation cost of high-quality multimodal instruction data remain major bottlenecks for multimodal model development. Raw videos contain rich visual knowledge and temporal information, but transforming them into accurate, verifiable, and task-oriented instruction data remains challenging.
To address this, we introduce VidUEU-Agent, an agentic data-curation framework that automatically constructs instruction data from raw videos for multimodal understanding, image editing, and unified understanding-and-generation tasks.
🔑 Key Highlights
🎬 Four-Stage Data Construction Pipeline:
VidUEU-Agent performs global perception to analyze video structure and filter low-quality shots, mines informative keyframes and multimodal signals, constructs structured metadata, and routes each metadata entry to suitable task types for instruction synthesis. Agent planning, tool use, and multi-stage quality control support the entire process.
🔍 Evidence-Grounded Data Synthesis:
The framework combines vision-language models with visual tools for quality assessment, semantic analysis, and information verification. During instruction synthesis, the model cites the evidence signals it uses, improving data traceability and factual consistency.
🔄 Autonomous Iteration with Training Feedback:
VidUEU-Agent uses downstream training and evaluation results to identify weaknesses and refine data distributions across capabilities, domains, and task types. Across four iterations, each using 30K samples to fine-tune BAGEL from the same initial checkpoint, performance improves overall on the three feedback benchmarks:
📦 VidUEU-100K Dataset:
Using this pipeline, we construct VidUEU-100K from 8,268 diverse videos, containing nearly 20,000 dense metadata annotations and 100,000 instruction samples: 40K understanding, 30K editing, and 30K unified samples, spanning nine task paradigms and five scenario categories.
Our work demonstrates how raw videos can be transformed into diverse, evidence-grounded instruction data through agent planning, tool augmentation, and training-feedback-driven refinement. We hope VidUEU-Agent and VidUEU-100K provide a scalable and reproducible foundation for developing unified multimodal models.
📄 Paper: https://arxiv.org/abs/2611.04418
💻 Code: https://github.com/LRuca/VidUEU-Agent
🤗 Dataset: https://huggingface.co/datasets/Lruca/VidUEU100k
We welcome feedback and discussions!
Source: Hugging Face Daily Papers · huggingface.co