Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Brief overview

Falcon is a causal audio-visual target speaker extraction model that extracts the voice of the target speaker from the mouth video of the target speaker in noisy overlapping speech, combined with a single multi-scale time-frequency separator and an encoder spatial speaker contrast loss, reducing interference speaker leakage without increasing inference cost.

Article

Falcon is a causal audio-visual target speaker extraction model that uses the target speaker's mouth video to extract their voice from noisy, overlapping speech. It combines a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that reduces interfering-speaker leakage without adding inference cost.

The paper also introduces RealSSA, a benchmark covering varied speaker counts, near- and far-field interference, reverberation, and sudden noise, with both simulated scenes and a held-out real-recorded evaluation set. Compared with causal AV-TFGridNet, Falcon reduces trainable parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× in the reported evaluation. Code, model weights, datasets, and interactive demos are available on the project page.