Blog

The latest from Google Research

Real-Time 3D Object Detection on Mobile Devices with MediaPipe

miércoles, 11 de marzo de 2020

Posted by Adel Ahmadyan and Tingbo Hou, Software Engineers, Google Research 2D object predictionMediaPipe

3D Object Detection from a single image. MediaPipe Objectron determines the position, orientation and size of everyday objects in real-time on mobile devices.

Obtaining Real-World 3D Training DataLIDARaugmented realityARCoreARKithundreds of millions3D point clouds

Real-world data annotation for 3D object detection. Right: 3D bounding boxes are annotated in the 3D world with detected surfaces and point clouds. Left: Projections of annotated 3D bounding boxes are overlaid on top of video frames making it easy to validate the annotation.

AR Synthetic Data Generation

An example of AR synthetic data generation. The virtual white-brown cereal box is rendered into the real scene, next to the real blue book.

An ML Pipeline for 3D Object Detectionbuilt a single-stage modelMobileNetv2EPnPa prioriAdreno 650 mobile GPU

Network architecture and post-processing for 3D object detection.

Sample results of our network — [left] original 2D image with estimated bounding boxes, [middle] object detection by Gaussian distribution, [right] predicted segmentation mask.

Detection and Tracking in MediaPipe2D object detection and trackinginstant motion trackingMotion StillsMediaPipeend-to-end demo mobile applicationshoeschairs

Examples of our 3D object detection in the wild.

AcknowledgementsThe research described in this post was done by Adel Ahmadyan, Tingbo Hou, Jianing Wei, Matthias Grundmann, Liangkai Zhang, Jiuqiang Tang, Chris McClanahan, Tyler Mullen, Buck Bourdon, Esha Uboweja, Mogan Shieh, Siarhei Kazakou, Ming Guang Yong, Chuo-Ling Chang, and James Bruce. We thank Aliaksandr Shyrokau and the annotation team for their diligence to high quality annotations.