New research on MLLM retrieval heads
The story so far
In 18 languages, visual languages, and full-modal models, the retrieval head remains sparse in the newly added modalities (such as audiovisual) and has a high degree of cross-modal sharing, with full-modal models mainly retaining the retrieval heads of their backbone models; removing these retrieval heads eliminates almost all audiovisual retrieval capabilities.
AI-written from the reports · Updated 20 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersA Broader Look into MLLM's Retrieval Heads
In 18 languages, visual languages, and full-modal models, the retrieval head remains sparse in the newly added modalities (such as audio-visual) and has a high degree of cross-modal sharing. Full-modal models mainly retain the backbone model's retrieval heads; removing these retrieval heads eliminates almost all audio-visual retrieval capabilities.