Skip to content
Live
Hugging Face Daily Papers·· 11 hours ago65

HiUni: Harmonizing Understanding and Generation in Shared Visual Space

HiUni: Harmonizing Understanding and Generation in Shared Visual Space

Brief overview

HiUni extends pre-trained visual language models into unified models by sharing language-aligned ViT and dense Transformer backbones, achieving visual understanding and generation capabilities. The model adopts Pixel Diffusion Head for pixel space optimization and introduces a lightweight HiGate path to provide fine spatial details, achieving average scores similar to pre-trained VLMs in understanding tasks, while also achieving competitive performance in image generation and editing.

Article

Published on Nov 19

Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Unified multimodal models aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of these two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained vision language model can be directly extended into a unified model with extensive parameter sharing. Specifically, HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side. Yet such a design introduces two key challenges. Generation in pixel space is difficult to optimize, and semantic abstraction in pretrained ViT discards fine spatial details required for synthesis. To address these challenges, HiUni introduces a Pixel Diffusion Head tailored to direct pixel-space optimization and a lightweight HiGate pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons and analyses further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.

View arXiv page View PDF Project page GitHub 7 Add to collection

Get this paper in your agent:

hf papers read 2611.26107

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

ernie-research/HiUni

Image-Text-to-Text •

5B •

Updated 29 minutes ago

•

1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.26107 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

Source: Hugging Face Daily Papers · huggingface.co