Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Published on Oct 29

Authors:

Abstract

Youtu-Parsing-Omni is a unified multi-modal parser that encodes documents, images, audio, and video into structured spatiotemporal records using shared weights and cross-modal fusion, achieving state-of-the-art results across parsing benchmarks through a self-evolving on-policy distillation procedure.

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

View arXiv page View PDF Project page Add to collection

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports).

Input	`--task`	`modality` / `subtype`	Key contents
Document page	`document`	`image` / `document`	layout elements with bbox, text / LaTeX / OTSL tables / Markdown charts / Mermaid flowcharts, reading order
Natural image	`natural_image`	`image` / `natural_image`	entities and text with bbox, tags, captions, global description
Chart	`graphics_chart`	`image` / `document`	one `chart` element: Markdown table, notes, caption
Flowchart	`graphics_flowchart`	`image` / `document`	one `flowchart` element: Mermaid, caption
Geometry figure	`graphics_geometric`	`image` / `document`	one `geometric` element: points, lines, arcs, shapes, geometric relations and measurements
Audio	`audio`	`audio` / –	vocal / non-vocal segments with timestamps, speakers, ASR, timbre / scene captions, acoustic events
Natural video	`natural_video`	`video` / `natural_video`	temporal segments with visual elements, actions, interactions, camera motion, audio track
Text-rich video	`textrich_video`	`video` / `text_rich_video`	segments with OCR + ASR and a Markdown `structured_report` of the whole video

Highlights:

Unified schema – one JSON envelope for seven parsing families, driven by the task prompt.
Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model.
Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results).
Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.