Blog

The latest news from Google AI

Leveraging Temporal Context for Object Detection

Friday, June 26, 2020

Posted by Sara Beery, Student Researcher and Jonathan Huang, Research Scientist, Google Researchmountain pass road conditionsecosystem phenologyleading to poor generalization to novel deploymentsLILA BCWildlife Insights

What’s in this image? Objects in images from static cameras can be very challenging to detect and categorize. Here, a foggy morning has made it very difficult to see a herd of wildebeest walking along the crest of a hill. [Image from Snapshot Serengeti]

Context R-CNN: Long Term Temporal Context for Per-Camera Object DetectionFaster R-CNNTF Object Detection API

Here, we can see how additional examples from the same scene help experts determine that the object is an animal and not background. Context such as the shape & size of the object, its attachment to a herd, and habitual grazing at certain times of day help determine that the species is a wildebeest. Useful examples occur throughout the month.

The Context R-CNN ModelFaster R-CNNmemory bankattention

High-level architecture diagram, showing how Context R-CNN incorporates long-term context within the Faster R-CNN model architecture.

similarity-based attentionmemory bankMM

Context R-CNN is able to leverage context (spanning up to 1 month) to correctly categorize the challenging wildebeest example we saw above. The green values are the corresponding attention weights for each boxed object.

Compared to a Faster R-CNN baseline (left), Context R-CNN (right) is able to capture challenging objects such as an elephant occluded by a tree, two poorly-lit impala, and a vervet monkey leaving the frame. [Images from Snapshot Serengeti]

ResultsSnapshot SerengetiCaltech Camera Trapsmean average precisionS3D

Comparison to a single frame Faster R-CNN baseline, showing both mean average precision (mAP) and average recall (AR) detection metrics.

Ongoing and Future WorkWildlife InsightsiWildCamCVPR Fine-Grained Visual Recognition WorkshopAcknowledgementsThis post reflects the work of the authors as well as the following group of core contributors: Vivek Rathod, Guanhang Wu, Ronny Votel. We are also grateful to Zhichao Lu, David Ross, Tanya Birch and the Wildlife Insights AI team, and Pietro Perona and the Caltech Computational Vision Lab.