Blog

The latest news from Google AI

Looking to Listen: Audio-Visual Speech Separation

Wednesday, April 11, 2018

Posted by Inbar Mosseri and Oran Lang, Software Engineers, Google Researchcocktail party effectLooking to Listen at the Cocktail PartySIGGRAPH 2018

The input to our method is a video with one or more people speaking, where the speech of interest is interfered by other speakers and/or background noise. The output is a decomposition of the input audio track into clean speech tracks, one for each person detected in the video.

An Audio-Visual Speech Separation ModelAudioSetspectrogramour paper

Our multi-stream, neural network-based model architecture.

Application to Speech Recognitionproject web pagethis work from UC Berkeleythis work from MITAcknowledgementsThe research described in this post was done by Ariel Ephrat (as an intern), Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, Bill Freeman and Michael Rubinstein. We would like to thank Yossi Matias and Google Research Israel for their support for the project, and John Hershey for his valuable feedback. We also thank Arkady Ziefman for his help with animations and figures, and Rachel Soh for helping us procure permissions for video content in our results.