Blog

The latest from Google Research

On-Device, Real-Time Hand Tracking with MediaPipe

Monday, August 19, 2019

Posted by Valentin Bazarevsky and Fan Zhang, Research Engineers, Google Researchsign languageaugmented realityCVPR 2019MediaPipeproviding this hand perception functionality

3D hand perception in real-time on a mobile phone via MediaPipe. Our solution uses machine learning to compute 21 3D keypoints of a hand from a video frame. Depth is indicated in grayscale.

An ML Pipeline for Hand Tracking and Gesture Recognition

A palm detector model (called BlazePalm) that operates on the full image and returns an oriented hand bounding box.

A hand landmark model that operates on the cropped image region defined by the palm detector and returns high fidelity 3D hand keypoints.

A gesture recognizer that classifies the previously computed keypoint configuration into a discrete set of gestures.

face meshML pipelinepose estimation

Hand perception pipeline overview.

BlazePalm: Realtime Hand/Palm Detectionsingle-shot detector modelBlazeFaceavailablenon-maximum suppression algorithmanchorsRetinaNetfocal lossaverage precisionHand Landmark Model

Top: Aligned hand crops passed to the tracking network with ground truth annotation. Bottom: Rendered synthetic hand images with ground truth annotation

Mixed training schema for hand tracking network. Cropped real-world photos and rendered synthetic images are used as input to predict 21 3D keypoints.

Mean regression error

Dataset

normalized by palm size

Only real-world

16.1 %

Only rendered synthetic

25.7 %

Mixed real-world + synthetic

13.4 %

Gesture Recognition

Implementation via MediaPipe MediaPipedirected graphwide variety of devices and platformsTFLite GPU inference

The hand landmark model’s output (REJECT_HAND_FLAG) controls when the hand detection model is triggered. This behavior is achieved by MediaPipe’s powerful synchronization building blocks, resulting in high performance and optimal throughput of the ML pipeline.

MediaPipehereFuture Directions

AcknowledgementsSpecial thanks to all our team members who worked on the tech with us: Andrey Vakunov, Andrei Tkachenka, Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, Kanstantsin Sokal‎, Buck Bourdon, Mogan Shieh, Ming Guang Yong, Anastasia Tkach, Jonathan Taylor, Sean Fanello, Sofien Bouaziz, Juhyun Lee‎, Chris McClanahan, Jiuqiang Tang‎, Esha Uboweja‎, Hadon Nash‎, Camillo Lugaresi, Michael Hays, Chuo-Ling Chang, Matsvei Zhdanovich and Matthias Grundmann.