Falcon 音视频目标说话人提取模型发布

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

要点速览

Falcon是一个因果音视频目标说话人提取模型，通过目标说话人嘴部视频从嘈杂重叠语音中提取其声音，结合单次多尺度时频分离器和编码器空间说话人对比损失，在不增加推理成本的情况下减少干扰说话人泄露。

正文

Falcon is a causal audio-visual target speaker extraction model that uses the target speaker's mouth video to extract their voice from noisy, overlapping speech. It combines a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that reduces interfering-speaker leakage without adding inference cost.

The paper also introduces RealSSA, a benchmark covering varied speaker counts, near- and far-field interference, reverberation, and sudden noise, with both simulated scenes and a held-out real-recorded evaluation set. Compared with causal AV-TFGridNet, Falcon reduces trainable parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× in the reported evaluation. Code, model weights, datasets, and interactive demos are available on the project page.