Google AI Blog

Turbo, An Improved Rainbow Colormap for Visualization

Tuesday, August 20, 2019

Posted by Anton Mikhailov, Senior Software Engineer, Daydreamdepth imagesimage differencinglinked to lower accuracysuch as medical imaging

Left: Disparity image displayed as greyscale. Right: The commonly used Jet rainbow map being used to create a false color image.

perceptually uniform

The above image with simulated Protanopia

many modern alternativesmatplotlib

Viridis

Inferno

hereherehereDevelopmentsRGBcubic spline

Screenshot of the interface used to create and tune Turbo.

C2 continuous

Turbo

Jet

Comparison with Common Color Maps, it does affect people’s choice

Turbo

Jet

Viridis

Inferno

Turbo

Jet

Viridis

Inferno

Lightness plots generated by converting the sRGB values to CIECAM02-UCS and displaying the lightness value (J) in greyscale. The black line traces the lightness value from the low end of the color map (left) to the high end (right).

achromatopsiaSemantic Layers

Turbo

Jet

Viridis

Inferno

Left: Quantized Turbo colormap. Up to 33 quantized colors remain distinguishable and smooth in both lightness and hue change. Right: Quantized Jet color map. Many neighboring colors appear the same; Yellow and Cyan colors appear brighter than the rest.

Quick Judgingour attention system prioritizes huedepth estimation

“Output” Viridis

“Ground Truth” Viridis

“Output” Turbo

“Ground Truth” Turbo

Diverging Map Use Casesdivergingdifference images

"Ground Truth" disparity image

Estimated disparity image

Difference Image (ground truth - estimated disparity image), visualized with Turbo

Accessibility for Color Blindnesscolor blindness simulator

Test Image

Blue cone monochromacy

Achromatopsia

Conclusionherehereherevisualizing disparity mapsAcknowledgementsAmbrus Csaszar stared at many color ramps with me in order to pick the right tradeoffs between uniformity and detail accentuation. Christian Haene integrated the map into our team’s tools, which caused wide usage and thus spurred further improvements. Matthias Kramm and Ruofei Du came up with closed form approximations.

On-Device, Real-Time Hand Tracking with MediaPipe

Monday, August 19, 2019

Posted by Valentin Bazarevsky and Fan Zhang, Research Engineers, Google Researchsign languageaugmented realityCVPR 2019MediaPipeproviding this hand perception functionality

3D hand perception in real-time on a mobile phone via MediaPipe. Our solution uses machine learning to compute 21 3D keypoints of a hand from a video frame. Depth is indicated in grayscale.

An ML Pipeline for Hand Tracking and Gesture Recognition

A palm detector model (called BlazePalm) that operates on the full image and returns an oriented hand bounding box.

A hand landmark model that operates on the cropped image region defined by the palm detector and returns high fidelity 3D hand keypoints.

A gesture recognizer that classifies the previously computed keypoint configuration into a discrete set of gestures.

face meshML pipelinepose estimation

Hand perception pipeline overview.

BlazePalm: Realtime Hand/Palm Detectionsingle-shot detector modelBlazeFaceavailablenon-maximum suppression algorithmanchorsRetinaNetfocal lossaverage precisionHand Landmark Model

Top: Aligned hand crops passed to the tracking network with ground truth annotation. Bottom: Rendered synthetic hand images with ground truth annotation

Mixed training schema for hand tracking network. Cropped real-world photos and rendered synthetic images are used as input to predict 21 3D keypoints.

Mean regression error

Dataset

normalized by palm size

Only real-world

16.1 %

Only rendered synthetic

25.7 %

Mixed real-world + synthetic

13.4 %

Gesture Recognition

Implementation via MediaPipe MediaPipedirected graphwide variety of devices and platformsTFLite GPU inference

The hand landmark model’s output (REJECT_HAND_FLAG) controls when the hand detection model is triggered. This behavior is achieved by MediaPipe’s powerful synchronization building blocks, resulting in high performance and optimal throughput of the ML pipeline.

MediaPipehereFuture Directions

AcknowledgementsSpecial thanks to all our team members who worked on the tech with us: Andrey Vakunov, Andrei Tkachenka, Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, Kanstantsin Sokal‎, Mogan Shieh, Ming Guang Yong, Anastasia Tkach, Jonathan Taylor, Sean Fanello, Sofien Bouaziz, Juhyun Lee‎, Chris McClanahan, Jiuqiang Tang‎, Esha Uboweja‎, Hadon Nash‎, Camillo Lugaresi, Michael Hays, Chuo-Ling Chang, Matsvei Zhdanovich and Matthias Grundmann.

Joint Speech Recognition and Speaker Diarization via Sequence Transduction

Friday, August 16, 2019

Posted by Laurent El Shafey, Software Engineer and Izhak Shafran, Research Scientist, Google Healthspeaker diarizationYesHave you been taking your heart medications regularly?Yes?This basic multi-stage approachrecurrent neural network transducerpresented recentlyJoint Speech Recognition and Speaker Diarization via Sequence TransductionInterspeech 2019Conventional Speaker Diarization SystemsGaussian mixture models

Conventional speaker diarization system infers speaker labels in the acoustic domain and then overlays the speaker labels on the words generated by a separate ASR system.

How often have you been taking the medication?When should we turn in the homework?recent blog postAn Integrated Speech Recognition and Speaker Diarization Systemencoder

An integrated speech recognition and speaker diarization system where the system jointly infers who spoke when and what.

loss functionforward-backward algorithmTPU friendly implementationefficient implementation of the RNN-T lossWhen is the homework due?I expect you to turn them in tomorrow before class

A comparison of errors committed by the conventional system vs. the RNN-T system, as categorized by human annotators.

understanding medical conversationsAcknowledgementsWe would like to thank Hagen Soltau without whose contributions this work would not have been possible. This work was performed in collaboration with Google Brain and Speech teams.

Blog

Turbo, An Improved Rainbow Colormap for Visualization

On-Device, Real-Time Hand Tracking with MediaPipe

Joint Speech Recognition and Speaker Diarization via Sequence Transduction

Labels

Archive

Feed