Blog

The latest from Google Research

Adding Sound Effect Information to YouTube Captions

jueves, 23 de marzo de 2017

Posted by Sourish Chaudhuri, Software Engineer, Sound UnderstandingYouTube has provided automatic caption trackswe announced

Click the CC button to see the sound effect captioning system in action.

Deep Neural NetworkA DNN Model for Ambient SoundDense Detections to CaptionsViterbi algorithm

(Left) The dense sequence of probabilities from our DNN for the occurrence over time of single sound category in a video. (Center) Binarized segments based on the modified Viterbi algorithm. (Right) The duration-based filter removes segments that are shorter in duration than desired for the class.

Adding Sound Effect Information into the Automatic Captions Track

Using separate parts of the screen for speech and sound effect captions.

Interleaving the speech and sound effect captions as they occur.

Only showing sound effect captions at the end of sentences or when there is a pause in speech (even if they occurred in the middle of speech).

How hearing users perceive captions when watching with the sound off.

Participants who could hear the audio were able to ignore the inaccuracies.

Participants who could not hear the audio interpreted the error as the presence of a sound event, and that they had not missed out on critical speech information.