Blog

The latest from Google Research

Real-Time AR Self-Expression with Machine Learning

viernes, 8 de marzo de 2019

Posted by Artsiom Ablavatski and Ivan Grishchenko, Research Engineers, Google AIAugmented realityAR features coming to Google MapsPlayground - a creative mode in the Pixel cameraYouTube StoriesARCoreAugmented Faces API

Our 3D mesh and some of the effects it enables

TensorFlow Liteits new mobile GPU functionalityYouTube Stories' latest ARCore SDKML Kit Face Contour Detection APIAn ML Pipeline for Selfie AR

Our 3D mesh in action

transfer learningMLKitreasonable

Dataset expansion and improvement pipeline

Hardware-tailored InferenceTensorFlow Litenewly introduced GPU back-end

Inference time per frame: CPU vs. GPU

Comparison of the most complex (left) and the lightest models (right). Temporal consistency as well as lip and eye tracking is slightly degraded on light models.

Simulating light reflections via environmental mapping for realistic rendering of glasses

Natural lighting by casting virtual object shadows onto the face mesh

Modelling face occlusions to hide virtual object parts behind a face, e.g. virtual glasses, as shown below.

YouTube Stories includes Creator Effects like realistic virtual glasses, based on our 3D mesh

Modelling Specular reflections applied on lips and

Face painting by using luminance-aware material

Case study comparing real make-up against our AR make-up on 5 subjects under different lighting conditions.

latest ARCore SDKAcknowledgementsWe would like to thank Yury Kartynnik, Valentin Bazarevsky, Andrey Vakunov, Siargey Pisarchyk, Andrei Tkachenka, and Matthias Grundmann for collaboration on developing the current mesh technology; Nick Dufour, Avneesh Sud and Chris Bregler for an earlier version of the technology based on parametric models; Kanstantsin Sokal, Matsvei Zhdanovich, Gregory Karpiak, Alexander Kanaukou, Suril Shah, Buck Bourdon, Camillo Lugaresi, Siarhei Kazakou and Igor Kibalchich for building the ML pipeline to drive impressive effects; Aleksandra Volf and the annotation team for their diligence and dedication to perfection; Andrei Kulik, Juhyun Lee, Raman Sarokin, Ekaterina Ignasheva, Nikolay Chirkov, and Yury Pisarchyk for careful benchmarking and insights on mobile GPU-centric network architecture optimizations.