Blog

The latest news from Google AI

Capturing Special Video Moments with Google Photos

Wednesday, April 3, 2019

Posted by Sudheendra Vijayanarasimhan and David Ross, Software EngineersRethinking the Faster R-CNN Architecture for Temporal Action Localizationtemporal action localization networkFaster R-CNN

An example of the detected action "blowing out candles"

Identifying Actions for Model TrainingComparison to Object Detectiontemporal action localization,object detectionfaster R-CNNfeatureconvolutional neural networkregion proposal network

Faster R-CNN architecture for object detection

Temporal Action Localizationsegment proposal network

Architecture for temporal action localization

Special Considerations for Temporal Action Localization

Actions have much larger variations in durations
The temporal extent of actions varies dramatically—from a fraction of a second to minutes. For long actions, it is not important to understand each and every frame of the action. Instead, we can get a better handle on the action by skimming quickly through the video, using dilated temporal convolutions. This approach allows TALNet to search the video for temporal patterns, while skipping over alternate frames based on a given dilation rate. Analysing the video with several different rates that are selected automatically according to the anchor segment's length enables efficient identification of actions as large as the entire video or as short as a second.

The context before and after an action are important
The moments preceding and following an action instance contain critical information for localization and classification, arguably more so than the spatial context of an object. Therefore, we explicitly encode the temporal context by extending the length of proposal segments on both the left and right by a fixed percentage of the segment's length in both the proposal generation stage and the classification stage.

Actions require multi-modal input
Actions are defined by appearance, motion and sometimes even audio information. Therefore, it is important to consider multiple modalities of features for the best results. We use a late fusion scheme for both the proposal generation network and the classification network, in which each modality has a separate proposal generation network whose outputs are combined together to obtain the final set of proposals. These proposals are classified using separate classification networks for each modality, which are then averaged to obtain the final predictions.

TALNet in ActionTHUMOS'14ActivityNet

An example of the detected action "sliding down a slide"

An example of the detected actions "jump into the pool" (left), "twirl in a dress" (center) and "feed baby a spoonful" (right).

Next stepsprecision and recallAcknowledgementsSpecial thanks Tim Novikoff and Yu-Wei Chao, as well as Bryan Seybold, Lily Kharevych, Siyu Gu, Tracy Gu, Tracy Utley, Yael Marzan, Jingyu Cui, Balakrishnan Varadarajan, Paul Natsev for their critical contributions to this project.