Google AI Blog

Building SMILY, a Human-Centric, Similar-Image Search Tool for Pathology

Friday, July 19, 2019

Posted by Narayan Hegde, Software Engineer, Google Health and Carrie J. Cai, Research Scientist, Google Researchdiabetic eye diseasemetastatic breast cancerwhathowanatomic pathologycontent-based image retrievalreverse image searchGoogle Imagessimilar image searchSimilar Image Search for Histopathology: SMILYNature Partner Journal (npj) Digital Medicine“Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making” herehonorable mention2019 ACM CHI Conference on Human Factors in Computing SystemsSMILY DesignembeddingThe Cancer Genome Atlas

Schematic of the steps in building the SMILY database and the process by which input image patches are used to perform the similar image search.

Example of selecting a small region in a slide and using SMILY to retrieve similar images. SMILY efficiently searches a database of billions of cropped images in a few seconds. Because pathology images can be viewed at different magnifications (zoom levels), SMILY automatically searches images at the same magnification as the input image.

Second example of using SMILY, this time searching for a lobular carcinoma, a specific subtype of breast cancer.

Refinement tools for SMILYlooks similariterative diagnosissecond paperrefine-by-region refine-by-examplerefine-by-concept

beyond

AcknowledgementsThis work would not have been possible without Jason D. Hipp, Yun Liu, Emily Reif, Daniel Smilkov, Michael Terry, Craig H. Mermel, Martin C. Stumpe and members of Google Health and PAIR. Preprints of the two papers are available here and here.

Parrotron: New Research into Improving Verbal Communication for People with Speech Impairments

Wednesday, July 17, 2019

Posted by Fadi Biadsy, Research Scientist and Ron Weiss, Software Engineer, Google Researchautomatic speech recognitiontext-to-speech synthesisProject EuphoniaParrotronend-to-enddeep neural network The Parrotron Speech Conversion Modelattention-based sequence-to-sequence modelParallel WaveNet. phonotacticProject Euphoniamultitask objectivespectrograms

Overview of the Parrotron model architecture. An input speech spectrogram is passed through encoder and decoder neural networks to generate an output spectrogram in a new voice.

Case Studiesprofoundly deaftransliterationpeople

Dimitri saying, "How far is the Moon from the Earth?"

Parrotron (male voice) saying, "How far are the Moon from the Earth?"

muscular dystrophyoutcomes

Aubrie saying, "Is morning glory a perennial plant?"

Parrotron (female voice) saying, "Is morning glory a perennial plant?"

Aubrie saying, "Schedule a meeting with John on Friday."

Parrotron (female voice) saying, "Schedule a meeting with John on Friday."

Cascaded ApproachProject Euphonia

Dimitri saying, "What is definition of rhythm?"

Parrotron (male voice) saying, "What is definition of rhythm?"

Dimitri saying, "How many ounces in one liter?"

Parrotron (male voice) saying, "Hey Google, How many unces [sic] in one liter?"

Google Assistant saying, "One liter is equal to thirty-three point eight one four US fluid ounces."

Aubrie saying, "Is it wheelchair accessible?"

Parrotron (female voice) saying, "Is it wheelchair accecable [sic]?"

Conclusionour papergithub repositorythis short form AcknowledgementsThis project was joint work between the Speech and Google Brain teams. Contributors include Fadi Biadsy, Ron Weiss, Pedro Moreno, Dimitri Kanevsky, Ye Jia, Suzan Schwartz, Landis Baker, Zelin Wu, Johan Schalkwyk, Yonghui Wu, Zhifeng Chen, Patrick Nguyen, Aubrie Lee, Andrew Rosenberg, Bhuvana Ramabhadran, Jason Pelecanos, Julie Cattiau, Michael Brenner, Dotan Emanuel, Joel Shor, Sean Lee and Benjamin Schroeder. Our data collection efforts have been vastly accelerated by our collaborations with ALS-TDI.

Multilingual Universal Sentence Encoder for Semantic Retrieval

Friday, July 12, 2019

Posted by Yinfei Yang and Amin Ahmad, Software Engineers, Google Researchintroduced last yearUniversal Sentence Encoder (USE) for EnglishTensorflow Hubsentiment classifierthree new USE multilingual modulesquestion-answer retrievalmulti-task dual-encoder frameworkdual-encoder with additive margin softmax approach

Multi-task training structure of the Universal Sentence Encoder. A variety of tasks and task structures are joined by shared encoder layers/parameters (pink boxes).

Semantic Retrieval Applicationsnearest neighbor problemprecision and recallapproximate

A prototypical semantic retrieval pipeline, used for textual similarity.

Semantic Similarity Modules

Multilingual Semantic Textual Similarity Retrieval
Most existing approaches for finding semantically similar text require being given a pair of texts to compare. However, using the Universal Sentence Encoder, semantically similar text can be extracted directly from a very large database. For example, in an application like FAQ search, a system can first index all possible questions with associated answers. Then, given a user’s question, the system can search for known questions that are semantically similar enough to provide an answer. A similar approach was used to find comparable sentences from 50 million sentences in wikipedia. With the new multilingual USE models, this can be done in any of supported non-English languages.

Multilingual Translation Pair Retrieval
The newly released modules can also be used to mine translation pairs to train neural machine translation systems. Given a source sentence in one language (“How do I get to the restroom?”), they can find the potential translation target in any other supported language (“¿Cómo llego al baño?”).

Chidambaram et al. (2018)USE for Question-Answer Retrievalquestion-answer retrieval applications

Visualizing the action of a neural answer retrieval system. The blue point at the north pole represents the question vector. The other points represent the embeddings of various answers. The correct answer, highlighted here in red, is “closest” to the question, in that it minimizes the angular distance. The points in this diagram are produced by an actual USE-QA model, however, they have been projected downwards from ℝ500 to ℝ3 to assist the reader’s visualization.

Google Talk to BooksWhat fragrance brings back memories?And for me, the smell of jasmine along with the pan bagnat, it brings back my entire carefree childhood.fragrancesmellFor Researchers and Developerssemantic similarity for natural languageAcknowledgementsMandy Guo, Daniel Cer, Noah Constant, Jax Law, Muthuraman Chidambaram for core modeling, Gustavo Hernandez Abrego, Chen Chen, Mario Guajardo-Cespedes for infrastructure and colabs, Steve Yuan, Chris Tar, Yunhsuan Sung, Brian Strope, Ray Kurzweil for discussion of the model architecture.

Advancing Semi-supervised Learning with Unsupervised Data Augmentation

Wednesday, July 10, 2019

Posted by Qizhe Xie, Student Researcher and Thang Luong, Senior Research Scientist, Google Research, Brain TeamGPUTPUImageNetnatural language processingvisionspeechsupervised learning

Example augmentation operations for text-based (top) or image-based (bottom) training data.

Unsupervised Data Augmentation (UDA) for Consistency Trainingsemi-supervised learningrevival of semi-supervised learningBERTgithubUnsupervised Data Augmentation Explainedloss functionconsistency training

An overview of Unsupervised Data Augmentation (UDA). Left: Standard supervised loss is computed when labeled data is available. Right: With unlabeled data, a consistency loss is computed between an example and its augmented version.

Gaussian noiseadversarial noiseback translationAutoAugmentTF-IDFBenchmarks in NLP and Computer VisionIMDb

Benchmark on IMDb, a sentiment analysis task. UDA surpasses state-of-the-art results in supervised learning across different training sizes.

CIFAR-10VATICTPyramidNet+ShakeDropSVHN

SSL benchmark on CIFAR-10, an image classification task. UDA surpases existing semi-supervised learning methods, all of which use the Wide-ResNet-28-2 architecture. UDA applies the augmentation policy found by AutoAugment using 4,000 examples. At this sample size, it matches the performance of the fully supervised setting with 50,000 examples. The same augmentation policy is applied in cases with fewer labeled examples.

ImageNetReleasecodebaseAcknowledgementsSpecial thanks to the co-authors of the paper Zihang Dai, Eduard Hovy, and Quoc V. Le. We’d also like to thank Hieu Pham, Adams Wei Yu, Zhilin Yang, Colin Raffel, Olga Wichrowska, Ekin Dogus Cubuk, Guokun Lai, Jiateng Xie, Yulun Du, Trieu Trinh, Ran Zhao, Ola Spyra, Brandon Yang, Daiyi Peng, Andrew Dai, Samy Bengio and Jeff Dean for their help with this project. A preprint is available online.

Predicting the Generalization Gap in Deep Neural Networks

Tuesday, July 9, 2019

Posted by Yiding Jiang, Google AI ResidentDeepneural networksimage recognitionimage segmentationmachine translationVC-dimensionRademacher complexitygeneralize poorlyrecent workmassivelygeneralization gapmargindecision boundarysupport-vector machinesmargintheoretical upper bounds

An example of a support-vector machine decision boundary. The hyperplane defined by w∙x-b=0 is the "decision boundary" of this linear classifier, i.e., every point x lying on the hyperplane is equally likely to be in either class under this classifier.

ICLR 2019Predicting the Generalization Gap in Deep Networks with Margin Distributionsnormalized margingeneralization gapGithub repository

Each plot corresponds to a convolutional neural network trained on CIFAR-10 with different classification accuracies. The probability density (y-axis) of normalized margin distributions (x-axis) at 4 layers of a network is shown for three different models with increasingly better generalization (left to right). The normalized margin distributions are strongly correlated with test accuracy, which suggests they can be used as a proxy for predicting a network's generalization gap. Please see our paper for more details on these networks.

Margin Distributions as a Predictor of Generalizationlinear regression

Predicted generalization gap (x-axis) vs. true generalization gap (y-axis) on CIFAR-100 + ResNet-32. The points lie close to the diagonal line, which indicates that the predicted values of the log linear model fit the true generalization gap very well.

The Deep Model Generalization DatasetDeep Model GeneralizationNetwork-in-NetworkregularizationGithub repository

Announcing the YouTube-8M Segments Dataset

Friday, June 28, 2019

Posted by Joonseok Lee and Joe Yue-Hei Ng, Software Engineers, Google ResearchFirstSecond YouTube-8M Large-Scale Video Understanding Challenge and WorkshopYouTube-8M SegmentsKaggle video understanding challenge3rd Workshop on YouTube-8M Large-Scale Video Understanding2019 International Conference on Computer Vision

YouTube-8M Segmentscapturing special video momentsYouTube-8M releaseThe 3rd YouTube-8M Video Understanding ChallengeKaggle competition pageThe 3rd Workshop on YouTube-8M Large-Scale Video Understandingworkshop pageAcknowledgementsThis post reflects the work of many machine perception researchers including Ke Chen, Nisarg Kothari, Joonseok Lee, Hanhan Li, Paul Natsev, Joe Yue-Hei Ng, Naderi Parizi, David Ross, Cordelia Schmid, Javier Snaider, Rahul Sukthankar, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan, Yexin Wang, Zheng Xu, as well as Julia Elliott and Walter Reade from Kaggle. We are also grateful for the support and advice from our partners at YouTube.

Predicting Bus Delays with Machine Learning

Thursday, June 27, 2019

Posted by Alex Fabrikant, Research Scientist, Google Researchintroduced live traffic delays for busesfirst launched in India three weeks agoThe Beginnings of a Model

speech processingmachine translation

To model a bus trip (a) starting at the blue stop, the model (b) adds up the delay predictions from timeline units for the blue stop, the three road segments, the white stop, etc.

Modeling the "Where"

popular times at businessesparking difficultyLearning the Local Rhythms

The model's time representation (3 out of 4 dimensions) forms a loop, reimagined here as the circumference of a watch. The more location-dependent time windows like 4pm-9pm and 7am-9am get more complex "crumpling", while big featureless windows like 2am-5am get bent away with flat bends for simpler rules. (Artist's conception by Will Cassella, using textures from textures.com and HDRIs from hdrihaven.)

Putting it All Together

The Trip AheadAcknowledgementsThis work was the joint effort of James Cook, Alex Fabrikant, Ivan Kuznetsov, and Fangzhou Xu, on Google Research, and Anthony Bertuca, Julian Gibbons, Thierry Le Boulengé, Cayden Meyer, Anatoli Plotnikov, and Ivan Volosyuk on Google Maps. We thank Senaka Buthpitiya, Da-Cheng Juan, Reuben Kan, Ramesh Nagarajan, Andrew Tomkins, and the greater Transit team for support and helpful discussions; as well as Will Cassella for the inspired reimagining of the model's time embedding. We are also indebted to our partner agencies for providing the transit data feeds the system is trained on.

Innovations in Graph Representation Learning

Tuesday, June 25, 2019

Posted by Alessandro Epasto, Senior Research Scientist and Bryan Perozzi, Senior Research Scientist, Graph Mining Teamgraph nodesedgesfriendship relationsgraph analysisclusteringlink predictionprivacyotherslearn. graph embeddingDeepWalk

Left: The well-known Karate graph representing a social network. Right: A continuous space embedding of the nodes in the graph using DeepWalk.

Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social ContextsWWW’19Watch Your Step: Learning Node Embeddings via Graph AttentionNeurIPS’18graph embeddingsLearning Node Representations that Capture Multiple Social Contextssinglesingleoverlappingmultiple WWW’19 paperSplitterego-network analysispersona graph conceptGPpersona graphGpersona

Ego-net of node U

personas

The ego-splitting method separating the U nodes in 2 personas.

up to 90% reduction

Top Left: A typical graphs with highly overlapping communities. Top Right: A traditional embedding of the graph on the left using node2vec. Bottom Left: A persona graph of the graph above. Bottom Right: The Splitter embedding of the persona graph. Notice how the persona graph clearly disentangles the overlapping communities of the original graph and Splitter outputs well-separated embeddings.

Automatic hyper-parameter tuning via graph attention.link prediction and node classificationsecond paper.DeepWalkbackpropagation

Our new method for automatic hyper-parameter tuning, Watch Your Step, uses an attention model to learn different graph context distributions. Shown above are two example local neighborhoods about a center node (in yellow) and the context distributions (red gradient) that was learned by the model. The left-side graph shows a more diffused attention model, while the distribution on the right shows one concentrated on direct neighbors.

AutoMLneural architecture searchcodeAcknowledgementsWe thank Sami Abu-el-Haija who contributed to this work and is now a Ph.D. student at USC. For more information on the Graph Mining team (part of Algorithm and Optimization), visit our pages.

Off-Policy Classification - A New Reinforcement Learning Model Selection Method

Wednesday, June 19, 2019

Reinforcement learningoff-policy RLwalkinggraspingfully off-policy RLentirelytrainingevaluationAutoMLff-policy evaluation

A diagram for real-world model development. Assuming we can evaluate 10 models per day, without off-policy evaluation, we would need 100x as many days to evaluate our models.

Off-Policy Evaluation via Off-Policy Classificationoff-policy classificationimportance samplingHow OPC WorkseffectivecatastrophicQ-functionQ-learningpositive-unlabeled learningOff-Policy Evaluation for Sim-to-Real Learningsimulated datatransfer learning techniquessample complexity

An example of how simulated experience can differ from real-world experience. Here, simulated images (left) have much less visual complexity than real-world images (right).

Results

An experiment in the simulated grasping task. The red curve is the dimensionless SoftOPC score over the course of training, evaluated from old data. The blue curve is the grasp success rate in simulation. We see the SoftOPC on old data correlates well with grasp success of the model within our simulator.

SoftOPC score and true performance for 3 different sim-to-real methods: a baseline simulation, a simulation with random textures and lighting, and a model trained with RCAN. All three models are trained with no real data, then evaluated with off-policy evaluation on a validation set of real data. The ordering of the SoftOPC score matches the order of real grasp success.

temporal-difference error

Results from our sim-to-real evaluation experiment. On the left is a baseline, the temporal difference error of the model. On the right is one of our proposed methods, the SoftOPC. The shaded region is a 95% confidence interval. The correlation is significantly better with SoftOPC.

Future WorkAcknowledgementsThis research was conducted by Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz and Sergey Levine. We’d like to thank Razvan Pascanu, Dale Schuurmans, George Tucker and Paul Wohlhart for valuable discussions. A preprint is available on arXiv.