Blog

The latest from Google Research

RepNet: Counting Repetitions in Videos

lunes, 22 de junio de 2020

Posted by Debidatta Dwibedi, Research Scientist, Robotics at GoogleCounting Out Time: Class Agnostic Video Repetition Counting in the WildRepNetour previous workdatasetColab notebookRepNetIn the pastResNetResNetimagevideotemporal self-similarity matrix

Demonstration of how the TSM processes images of the Earth’s day-night cycle.

Transformers

Overview of the RepNet model.

Temporal Self-Similarity Matrix

Jumping Jacks (constant period; video from Kinetics), Bouncing ball (decreasing period; Kinetics), Mixing concrete (aperiodic segments present in video; PERTUBE dataset).

Data

Our synthetic data generation pipeline that produces videos with repetitions from any video.

camera motion augmentationaffine motion

Left: An example of a synthetic repeating video generated from a random video. Right: An example of a video with camera motion augmentation, which is tougher for the model, but results in better generalization to real repeating videos (both from Kinetics).

EvaluationKinetics datasetCountixApplications

RepNet can count repeated activities from a range of domains, such as slicing onions (left; video from Kinetics dataset), Earth’s diurnal cycle (middle; Himawari satellite data), or even a cheetah in motion (right; video from imgur.com).

Predicted heart rates: 45 bpm (left) and 75 bpm (right). True heart rates 46-50 bpm and 78-79 bpm, respectively. RepNet’s prediction of the heart rate across different devices is encouragingly close to the rate measured by the device. (Source for left and right)

In this video, we see RepNet counting accelerating cellular oscillations observed under a laser microscope even though it has never seen such a video during training, (from Nature article).

Left: Person performing a “mountain climber” exercise. Right: The 1D projection of the RepNet embeddings using principal component analysis, capturing the moment that the person changes their speed during the exercise. (Video from Kinetics)

ReleaseannotationsColab notebookAcknowledgementsThis is joint work with Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Special thanks to Tom Small for designing the visual explanation of TSM. The authors thank Anelia Angelova, Relja Arandjelović, Sourish Chaudhuri, Aishwarya Gomatam, Meghana Thotakuri, and Vincent Vanhoucke for their help with this project.