Blog
The latest news from Google AI
Joint Speech Recognition and Speaker Diarization via Sequence Transduction
Friday, August 16, 2019
Posted by Laurent El Shafey, Software Engineer and Izhak Shafran, Research Scientist, Google Health
Being able to recognize “who said what,” or
speaker diarization
, is a critical step in understanding audio of human dialog through automated means. For instance, in a medical conversation between doctors and patients, “
Yes
” uttered by a patient in response to “
Have you been taking your heart medications regularly?
” has a substantially different implication than a rhetorical “
Yes?
” from a physician.
Conventional speaker diarization (SD) systems use two stages, the first of which detects changes in the acoustic spectrum to determine when the speakers in a conversation change, and the second of which identifies individual speakers across the conversation.
This basic multi-stage approach
is almost two decades old, and during that time only the speaker change detection component has improved.
With the recent development of a novel neural network model—the
recurrent neural network transducer
(RNN-T)—we now have a suitable architecture to improve the performance of speaker diarization addressing some of the limitations of the previous diarization system we
presented recently
. As reported in our recent paper, “
Joint Speech Recognition and Speaker Diarization via Sequence Transduction
,” to be presented at
Interspeech 2019
, we have developed an RNN-T based speaker diarization system and have demonstrated a breakthrough in performance from about 20% to 2% in word diarization error rate—a factor of 10 improvement.
Conventional Speaker Diarization Systems
Conventional speaker diarization systems rely on differences in how people sound acoustically to distinguish the speakers in the conversations. While male and female speakers can be identified relatively easily from their pitch using simple acoustic models (e.g.,
Gaussian mixture models
) in a single stage, speaker diarization systems use a multi-stage approach to distinguish between speakers having potentially similar pitch. First, a change detection algorithm breaks up the conversation into homogeneous segments, hopefully containing only a single speaker, based upon detected vocal characteristics. Then, deep learning models are employed to map segments from each speaker to an embedding vector. Finally, in a clustering stage, these embeddings are grouped together to keep track of the same speaker across the conversation.
In practice, the speaker diarization system runs in parallel to the automatic speech recognition (ASR) system and the outputs of the two systems are combined to attribute speaker labels to the recognized words.
Conventional speaker diarization system infers speaker labels in the acoustic domain and then overlays the speaker labels on the words generated by a separate ASR system.
There are several limitations with this approach that have hindered progress in this field. First, the conversation needs to be broken up into segments that only contain speech from one speaker. Otherwise, the embedding will not accurately represent the speaker. In practice, however, the change detection algorithm is imperfect, resulting in segments that may contain multiple speakers. Second, the clustering stage requires that the number of speakers be known and is particularly sensitive to the accuracy of this input. Third, the system needs to make a very difficult trade-off between the segment size over which the voice signatures are estimated and the desired model accuracy. The longer the segment, the better the quality of the voice signature, since the model has more information about the speaker. This comes at the risk of attributing short interjections to the wrong speaker, which could have very high consequences, for example, in the context of processing a clinical or financial conversation where affirmation or negation needs to be tracked accurately. Finally, conventional speaker diarization systems do not have an easy mechanism to take advantage of linguistic cues that are particularly prominent in many natural conversations. An utterance, such as “
How often have you been taking the medication?
” in a clinical conversation is most likely uttered by a medical provider, not a patient. Likewise, the utterance, “
When should we turn in the homework?
” is most likely uttered by a student, not a teacher. Linguistic cues also signal high probability of changes in speaker turns, for example, after a question.
There are a few exceptions to the conventional speaker diarization system, and one such exception was reported in our
recent blog post
. In that work, the hidden states of the recurrent neural network (RNN) tracked the speakers, circumventing the weakness of the clustering stage. The work reported in this post takes a different approach and incorporates linguistic cues, as well.
An Integrated Speech Recognition and Speaker Diarization System
We developed a novel and simple model that not only combines acoustic and linguistic cues seamlessly, but also combines speaker diarization and speech recognition into one system. The integrated model does not degrade the speech recognition performance significantly compared to an equivalent recognition only system.
The key insight in our work was to recognize that the RNN-T architecture is well-suited to integrate acoustic and linguistic cues. The RNN-T model consists of three different networks: (1) a transcription network (or
encoder
) that maps the acoustic frames to a latent representation, (2) a prediction network that predicts the next target label given the previous target labels, and (3) a joint network that combines the output of the previous two networks and generates a probability distribution over the set of output labels at that time step. Note, there is a feedback loop in the architecture (diagram below) where previously recognized words are fed back as input, and this allows the RNN-T model to incorporate linguistic cues, such as the end of a question.
An integrated speech recognition and speaker diarization system where the system jointly infers who spoke when and what.
Training the RNN-T model on accelerators like graphical processing units (GPU) or tensor processing units (TPU) is non-trivial as computation of the
loss function
requires running the
forward-backward algorithm
, which includes all possible alignments of the input and the output sequences. This issue was addressed recently in a
TPU friendly implementation
of the forward-backward algorithm, which recasts the problem as a sequence of matrix multiplications. We also took advantage of an
efficient implementation of the RNN-T loss
in TensorFlow that allowed quick iterations of model development and trained a very deep network.
The integrated model can be trained just like a speech recognition system. The reference transcripts for training contain words spoken by a speaker followed by a tag that defines the role of the speaker. For example, “
When is the homework due?
” ≺student≻, “
I expect you to turn them in tomorrow before class
,” ≺teacher≻. Once the model is trained with examples of audio and corresponding reference transcripts, a user can feed in the recording of the conversation and expect to see an output in a similar form. Our analyses show that improvements from the RNN-T system impact all categories of errors, including short speaker turns, splitting at the word boundaries, incorrect speaker assignment in the presence of overlapping speech, and poor audio quality. Moreover, the RNN-T system exhibited consistent performance across conversation with substantially lower variance in average error rate per conversation compared to the conventional system.
A comparison of errors committed by the conventional system vs. the RNN-T system, as categorized by human annotators.
Furthermore, this integrated model can predict other labels necessary for generating more reader-friendly ASR transcripts. For example, we have been able to successfully improve our transcripts with punctuation and capitalization symbols using the appropriately matched training data. Our outputs have lower punctuation and capitalization errors than our previous models that were separately trained and added as a post-processing step after ASR.
This model has now become a standard component in our project on
understanding medical conversations
and is also being adopted more widely in our non-medical speech services.
Acknowledgements
We would like to thank Hagen Soltau without whose contributions this work would not have been possible. This work was performed in collaboration with Google Brain and Speech teams.
Project Euphonia’s Personalized Speech Recognition for Non-Standard Speech
Tuesday, August 13, 2019
Posted by Joel Shor and Dotan Emanuel, Research Engineers, Google Research, Tel Aviv
The utility of technology is dependent on its accessibility. One key component of accessibility is automatic speech recognition (ASR), which can greatly improve the ability of those with speech impairments to interact with every-day smart devices. However, ASR systems are most often trained from 'typical' speech, which means that underrepresented groups, such as those with speech impairments or heavy accents, don't experience the same degree of utility. For example, amyotrophic lateral sclerosis (ALS) is a disease that can adversely affect a person’s speech—about 25% of people with ALS experiencing slurred speech as their first symptom. In addition, most people with ALS eventually lose the ability to walk, so being able to interact with automated devices from a distance can be very important. Yet current state-of-the-art ASR models can yield high
word error rates
(WER) for speakers with only a moderate speech impairment from ALS, effectively barring access to ASR reliant technologies.
In “
Personalizing ASR for Dysarthric and Accented Speech with Limited Data
,” to be presented at
Interspeech 2019
, we describe some of the research behind
Project Euphonia
, an ASR platform that performs speech-to-text transcription. This work presents an approach to improve ASR for people with ALS that may also be applicable to many other types of non-standard speech. Using a two-step training approach that starts with a baseline “standard” corpus and then fine-tunes the training with a personalized speech dataset, we have demonstrated significant improvements for speakers with atypical speech over current state-of-the-art models.
A Two-Phased Approach to Training
In order to create ASR models that work on non-standard speech, one needs to overcome two challenges. The first is that within a particular class of atypical speech, be it a regional accent or a speech impairment, for example, individuals can exhibit very different ways of speaking. Our approach deals with this sub-group heterogeneity by training the ASR model in two phases. We start with a high-quality ASR model trained on thousands of hours of standard speech and then we fine-tune parts of the model to an individual with non-standard speech. This approach is similar to that of
Parrotron
: both systems use end-to-end neural networks to help improve communication and accessibility, but Parrotron focuses exclusively on speech-to-speech, where a person’s speech is converted directly into synthesized speech, rather than text.
The second challenge arises from the difficulty in collecting enough data to train a state-of-the-art recognizer for individuals. Typical speech recognizers are trained on thousands of hours of speech from many different speakers. Acquiring this much data from a single speaker is nearly impossible, especially if the speaker may experience exhaustion from speaking due to a medical condition. Our approach overcomes this issue by first training a base model on a large corpus of typical speech, and then training a personalized model using a much smaller dataset with the targeted non-standard speech characteristics.
The Neural Network Architecture
When developing the models used for training data on atypical speech, we explored two different neural architectures. The first is the
RNN-Transducer
(RNN-T), a neural network architecture consisting of encoder and decoder networks that has shown good results on numerous ASR tasks. The encoder is bidirectional (i.e., it looks at the entire sentence at once in order to provide context), and thus it requires the entire audio sample to perform speech recognition.
The other architecture we explored was
Listen, Attend, and Spell
(LAS), which is an attention-based, sequence-to-sequence model that maps sequences of acoustic properties to sequences of languages. This model uses an encoder to convert the sequence of acoustic frames to a sequence of internal representations, and a decoder to convert the sequence of internal representations to linguistic output. The network produces “word pieces”, which are a linguistic representation between
graphemes
and words.
Comparison of the RNN-Transducer (left) and Listen, Attend, Spell (right) architectures. From
Prabhavalkar et al. 2017
.
We experimented with fine-tuning the state-of-the-art RNN-T and LAS base models on two types of non-standard speech. In partnership with the
ALS Therapy Development Institute
, we first collected about 36 hours of audio from 67 speakers who have ALS. The participants recorded themselves on their home computers using custom software while they read sentences from a very restricted language domain. Many phrases were single sentences with simple grammatical structure (e.g., “
What time is the basketball game on tonight?
”). This is in contrast with unrestricted language domains, which include domain-specific vocabulary (e.g., science talks) and complex language structure (e.g., a debate). The recordings did not include many of the
filler words
common in normal speech, such as “
um
” and “
uh
”.
We also tested accented speech, using the open source
L2 Arctic dataset
of non-native speech, which consists of 20 speakers with approximately 1 hour of speech per speaker. Each speaker recorded a set of 1150 utterances from the
CMU Arctic prompts
.
Audio
Euphonia Model
Standard Speech Model
Did I have anything to say about it?
Dictatorship angels
to
think
about it
Come right back please
Cameras object
Let’s try that again
It extracts
Turn it down a little bit please
Turning
down a little bit please
The audio (
left
) are recordings of a speaker with ALS. The text transcriptions are output from the Euphonia model (
center
) and the Standard Speech model (
right
). Incorrectly transcribed text is underlined.
Results
The absolute word error rates on the language-restricted test set is shown below. There is an improvement over the baseline model for very non-standard speech (heavy accents and ALS speech below 3 on the
ALS Functional Rating Scale
) and moderate improvements in ALS speech that is similar to typical speech. The relative difference between the base model and the fine-tuned model demonstrates that the majority of the improvement comes from the fine-tuning process, except in the case of the RNN-T on the Arctic dataset, where the RNN-T baseline is already strong.
1
Non-native English speech from the L2-Arctic dataset.
2
Low FRS (ALS Functional Rating Scale) speech; intelligible with repeating (FRS 2); Speech combined with non-vocal communication (FRS 1).
3
FRS 3; detectable speech disturbance.
The RNN-T model achieved 91% of the improvement by fine-tuning just two layers, most of which are close to the input. On the accented dataset, fine-tuning the same two layers achieved 86% of the relative improvement compared to fine-tuning the entire network. This is consistent with
previous
speech
work
.
Most of the performance gains were achieved early in training. The models we trained were tested on a relatively limited domain of vocabulary and linguistic complexity, so the performance numbers are not necessarily related to how well the models perform on more general tasks. We hope that just fine-tuning part of the network allows it to retain the acoustic and linguistic information from the general speech model, while needing minimal modifications to adapt to a single new speaker. Future work will test this hypothesis.
Low FRS corresponds to the ALS speakers with low intelligibility (FRS 2, 1), while high FRS corresponds to ALS speakers with less severely impacted speech (FRS 3).
Understanding Model Behavior
To better understand how our models improved after fine-tuning, we looked at the pattern of phoneme mistakes. We started by comparing the distribution of phoneme mistakes made by the base ASR model on standard speech to the mistakes made on ALS speech. The
SAMPA phonemes
with the five largest differences between the ALS data and standard speech are
p
,
U
,
f
,
k
, and
Z,
which
account for 20% of the deletion mistakes. Similarly, the
n
and
m
phonemes together account for 17% of the insertion / substitution mistakes. The same analysis on our fine-tuned models verifies that the unrecognized phoneme distribution is more similar to that of standard speech.
Our analysis shows that there are two aspects to every mistake: which phoneme the system doesn’t understand, and which phoneme the system
thinks
was said. Imagine having two systems with identical accuracy: one system always thinks that the
f
phoneme is actually the
g
phoneme, while another doesn't know what the
f
phoneme is and randomly guesses. These two systems will have identical performance and identical distributions of phoneme mistakes, but very different distributions of the predicted phoneme when a mistake is made. Surprisingly, ASR mistakes on ALS speech are far more similar to regular speech mistakes after Euphonia fine-tuning.
Deletion / substitution mistakes per
SAMPA phoneme
on ALS speech before fine-tuning, ALS speech after fine-tuning, and on typical speech (
Librispeech
dataset).
Future Work
In the future, we intend to explore additional techniques that can be helpful in the low data regime. We also hope to use phoneme mistakes to weight certain examples during training, or to pick training sentences for people with ALS to record that contain the most common phoneme mistakes. We would like to explore pooling data from multiple speakers with similar conditions.
We hope that continued research in this area will help voice interfaces become accessible to more people, especially those who need it most. One key component to this is collecting data. Anyone 18 or older can help us build better personalized models by donating audio data. If you’re interested, you can fill out
this form
to allow Google to contact you.
Acknowledgements
This work would not have been possible without the extraordinary effort and support of the ALS Therapy Development Institute and the ALS community, especially Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, and the individuals with ALS who kindly and patiently volunteered their audio. This work builds on the pioneering advances in speech recognition made by Google's speech team, in particular the recent development and deployment of end-to-end speech recognition models. We are grateful to the Google speech team for advice and collaboration, particularly to Anshuman Tripathi and Hasim Sak who guided us in training the initial models. We’d also like to thank Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Tara Sainath, Ding Zhao, Qiao Liang, Chung-Cheng Chiu, Dan Liebling, Ron Weiss, Anjuli Kannan, Dimitri Kanevsky, Ryan He, Gabor Simko, Benjamin Lee, Françoise Beaufays, Khe Chai Sim, Jimmy Tobin, Chet Gnegy, Jacqueline Huang, Ye Jia, Yu Zhang, Yonghui Wu, Michelle Ramanovich, Rus Heywood, Katrin Tomanek, Bob MacDonald, Pan-Pan Jiang, Ronnie Maor, Rif A. Saurous, Trevor Strohman, Dick Lyon, Avinatan Hassidim, Philip Nelson, and Yossi Matias for their technical contributions and project guidance.
Video Understanding Using Temporal Cycle-Consistency Learning
Thursday, August 8, 2019
Posted by Debidatta Dwibedi, Research Associate, Google Research
In the last few years there has been great progress in the field of video understanding. For example, supervised learning and
powerful
deep
learning
models
can be used to classify a number of possible actions in videos, summarizing the entire clip with a single label. However, there exist many scenarios in which we need more than just one label for the entire clip. For example, if a robot is pouring water into a cup, simply recognizing the action of “pouring a liquid” is insufficient to predict when the water will overflow. For that, it is necessary to track frame-by-frame the amount of water in the cup as it is being filled. Similarly, a baseball coach who is comparing stances of pitchers may want to retrieve video frames from the precise moment that the ball leaves the pitchers’ hands. Such applications require models to understand each
frame
of a video.
However, applying supervised learning to understand each individual frame in a video is expensive, since per-frame labels in videos of the action of interest are needed. This requires that annotators apply fine-grained labels to videos by manually adding unambiguous labels to every frame in each video. Only then can the model be trained, and only on a single action. Training on new actions requires the process to be repeated. With the increasing demand for fine-grained labeling, necessary for applications ranging from robotics to sports analytics, this makes the need for scalable learning algorithms that can understand videos without the tedious labeling process increasingly pertinent.
We propose a potential solution using a self-supervised learning method called
Temporal Cycle-Consistency Learning
(TCC). This novel approach uses correspondences between examples of similar sequential processes to learn representations particularly well-suited for fine-grained temporal understanding of videos. We are also releasing our TCC
codebase
to enable end-users to apply our self-supervised learning algorithm to new and novel applications.
Representation Learning Using TCC
A plant growing from a seedling to a tree; the daily routine of getting up, going to work and coming back home; or a person pouring themselves a glass of water are all examples of events that happen in a particular order. Videos capturing such processes provide temporal correspondences across multiple instances of the same process. For example, when pouring a drink one could be reaching for a teapot, a bottle of wine, or a glass of water to pour from. Key moments are common to all pouring videos (e.g., the first touch to the container or the container being lifted from the ground) and exist independent of many varying factors, such as visual changes in viewpoint, scale, container style, or the speed of the event. TCC attempts to find such correspondences across videos of the same action by leveraging the principle of cycle-consistency, which has been applied successfully in many problems in
computer
vision
, to learn useful visual representations by aligning videos.
The objective of this training algorithm is to learn a frame encoder, using any network architecture that processes images, such as
ResNet
. To do so, we pass all frames of the videos to be aligned through the encoder to produce their corresponding embeddings. We then select two videos for TCC learning, say
video 1
(the reference video) and
video 2
. A reference frame is chosen from
video 1
and its nearest neighbor frame (
NN
2
) from
video 2
is found in the embedding space (not pixel space). We then cycle back by finding the nearest neighbor of
NN
2
in
video 1
, which we call
NN
1
. If the representations are cycle-consistent, then the nearest neighbor frame in
video 1
(
NN
1
) should refer back to the starting reference frame.
We train the embedder using the distance between the starting reference frame and
NN
1
as the training signal. As training proceeds, the embeddings improve and reduce the cycle-consistency loss by developing a semantic understanding of each video frame in the context of the action being performed.
Using TCC, we learn embeddings with temporally fine-grained understanding of an action by aligning related videos.
What Does TCC Learn?
In the following figure, we show a model trained using TCC on
videos from the Penn Action Dataset
of people performing squat exercises. Each point on the left corresponds to frame embeddings, with the highlighted points tracking the embedding of the current video frame. Notice how the embeddings move collectively in spite of many differences in pose, lighting, body and object type. TCC embeddings encode the different phases of squatting without being provided explicit labels.
Right:
Input videos of people performing a squat exercise. The video on the top left is the reference. The other videos show nearest neighbor frames (in the TCC embedding space) from other videos of people doing squats.
Left:
The corresponding frame embeddings move as the action is performed.
Applications of TCC
The learned per-frame embeddings enable an array of interesting applications:
Few-shot action phase classification
When few labeled videos are available for training, the
few-shot
scenario, TCC performs very well. In fact, TCC can classify the phases of different actions with as few as a single labeled video. In the next figure we compare to other supervised and
self-supervised
learning
approaches in the few-shot setting. We find that supervised learning requires about 50 videos with each frame labeled to achieve the same accuracy that self-supervised methods achieve with just one fully labeled video.
Comparison of self-supervised and supervised learning for few-shot action phase classification.
Unsupervised video alignment
Aligning or synchronizing videos manually becomes prohibitively difficult as the number of videos increases. Using TCC, many videos can be aligned by selecting the nearest neighbor to each frame in a reference video, without the need for additional labels, as demonstrated in the figure below.
Results of unsupervised video alignment on videos of people pitching baseball using the distance between frames in the TCC space. The reference video used for alignment is shown in the upper left panel.
Label/modality transfer between videos
Just as TCC finds similar frames by using a nearest neighbor search in the embedding space, it can transfer metadata associated with any frame in one video to its matching frame in another video. This metadata can be in the form of temporal semantic labels or other modalities, such as sound or text. In the video below we show two examples where we can transfer the sound of liquid being poured into a cup from one video to another.
Per-frame Retrieval
With TCC, each frame in a video can be used as a query for retrieval of similar frames by looking up the nearest neighbors in the learned embedding space. The embeddings are powerful enough to differentiate between frames that look quite similar, such as frames just before or after the release of a bowling ball.
We can perform retrieval from videos on a per-frame basis, i.e., any frame can be used to look up similar frames in a large collection of videos. The retrieved nearest neighbors show that the model captures fine-grained differences in the scene.
Release
We are releasing our
codebase
, which includes implementations of a number of state-of-the-art self-supervised learning methods, including TCC. This codebase will be useful for researchers working on video understanding, as well as artists looking to use machine learning to align videos to create mosaics of people, animals, and objects moving synchronously.
Acknowledgements
This is joint work with Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. The authors would like to thank Alexandre Passos, Allen Lavoie, Anelia Angelova, Bryan Seybold, Priya Gupta, Relja Arandjelović, Sergio Guadarrama, Sourish Chaudhuri, and Vincent Vanhoucke for their help with this project. The videos used in this project come from the
PennAction
dataset. We thank the creators of PennAction for curating such an interesting dataset.
EfficientNet-EdgeTPU: Creating Accelerator-Optimized Neural Networks with AutoML
Tuesday, August 6, 2019
Posted by Suyog Gupta, Machine Learning Accelerator Architect and Mingxing Tan, Software Engineer, Google Research
For several decades, computer processors have doubled their performance every couple of years by reducing the size of the transistors inside each chip, as described by
Moore’s Law
. As reducing transistor size becomes more and more difficult, there is a renewed focus in the industry on developing domain-specific architectures — such as
hardware accelerators
— to continue advancing computational power. This is especially true for machine learning, where efforts are aimed at building specialized architectures for neural network (NN) acceleration. Ironically, while there has been a steady proliferation of these architectures in data centers and on
edge computing
platforms, the NNs that run on them are rarely customized to take advantage of the underlying hardware.
Today, we are happy to announce the release of
EfficientNet-EdgeTPU
, a family of image classification models derived from
EfficientNets
, but customized to run optimally on Google’s
Edge TPU
, a power-efficient
hardware accelerator available to developers through the
Coral Dev Board
and a
USB Accelerator
. Through such model customizations, the Edge TPU is able to provide real-time image classification performance while simultaneously achieving accuracies typically seen only when running much larger, compute-heavy models in data centers.
Using AutoML to customize EfficientNets for Edge TPU
EfficientNets
have been shown to achieve state-of-the-art accuracy in image classification tasks while significantly reducing the model size and computational complexity. To build EfficientNets designed to leverage the Edge TPU’s accelerator architecture, we invoked the
AutoML MNAS framework
and augmented the original EfficientNet’s neural network architecture search space with building blocks that execute efficiently on the Edge TPU (discussed below). We also built and integrated a “latency predictor” module that provides an estimate of the model latency when executing on the Edge TPU, by running the models on a cycle-accurate architectural simulator. The AutoML MNAS controller implements a
reinforcement learning
algorithm to search this space while attempting to maximize the reward, which is a joint function of the predicted latency and model accuracy. From past experience, we know that Edge TPU’s power efficiency and performance tend to be maximized when the model fits within its on-chip memory. Hence we also modified the reward function to generate a higher reward for models that satisfy this constraint.
Overall AutoML flow for designing customized EfficientNet-EdgeTPU models.
Search Space Design
When performing the architecture search described above, one must consider that EfficientNets rely primarily on
depthwise-separable convolutions
, a type of neural network block that factorizes a regular convolution to reduce the number of parameters as well as the amount of computations. However, for certain configurations, a regular convolution utilizes the Edge TPU architecture more efficiently and executes faster, despite the much larger amount of compute. While it is possible, albeit tedious, to manually craft a network that uses an optimal combination of the different building blocks, augmenting the AutoML search space with these accelerator-optimal blocks is a more scalable approach.
A regular 3x3 convolution (right) has more compute (multiply-and-accumulate (mac) operations) than an depthwise-separable convolution (left), but for certain input/output shapes, executes faster on Edge TPU due to ~3x more effective hardware utilization.
In addition, removing certain operations from the search space that require modifications to the Edge TPU compiler to fully support, such
swish non-linearity
and
squeeze-and-excitation block
, naturally leads to models that are readily ported to the Edge TPU hardware. These operations tend to improve model quality slightly, so by eliminating them from the search space, we have effectively instructed AutoML to discover alternate network architectures that may compensate for any potential loss in quality.
Model Performance
The neural architecture search (NAS) described above produced a baseline model,
EfficientNet-EdgeTPU-S
, which is subsequently scaled up using EfficientNet's
compound scaling
method to produce the
-M
and
-L
models. The compound scaling approach selects an optimal combination of input image resolution scaling, network width, and depth scaling to construct larger, more accurate models. The
-M
, and
-L
models achieve higher accuracy at the cost of increased latency as shown in the figure below.
EfficientNet-EdgeTPU-S/M/L models achieve better latency and accuracy than existing EfficientNets (B1),
ResNet
, and
Inception
by specializing the network architecture for Edge TPU hardware. In particular, our EfficientNet-EdgeTPU-S achieves higher accuracy, yet runs 10x faster than ResNet-50.
Interestingly, the NAS-generated model employs the regular convolution quite extensively in the initial part of the network where the depthwise-separable convolution tends to be less effective than the regular convolution when executed on the accelerator. This clearly highlights the fact that trade-offs usually made while optimizing models for general purpose CPUs (reducing the total number of operations, for example) are not necessarily optimal for hardware accelerators. Also, these models achieve high accuracy even without the use of esoteric operations. Comparing with the other image classification models such as Inception-resnet-v2 and Resnet50, EfficientNet-EdgeTPU models are not only more accurate, but also run faster on Edge TPUs.
This work represents a first experiment in building accelerator-optimized models using AutoML. The AutoML-based model customization can be extended to not only a wide range of hardware accelerators, but also to several different applications that rely on neural networks.
From Cloud TPU training to Edge TPU deployment
We have released the training code and pretrained models for EfficientNet-EdgeTPU on our
github repository
. We employ tensorflow’s
post-training quantization tool
to convert a floating-point trained model to an Edge TPU-compatible integer-quantized model. For these models, the post-training quantization works remarkably well and produces only a very slight loss in accuracy (~0.5%). The script for exporting the quantized model from a training checkpoint can be found
here
. For an update on the Coral platform, see
this post on the Google Developer’s Blog
, and for full reference materials and detailed instructions, please refer to the
Coral website
.
Acknowledgements
Special thanks to Quoc Le, Hongkun Yu, Yunlu Li, Ruoming Pang, and Vijay Vasudevan from the Google Brain team; Bo Wu, Vikram Tank, and Ajay Nair from the Google Coral team; Han Vanholder, Ravi Narayanaswami, John Joseph, Dong Hyuk Woo, Raksit Ashok, Jason Jong Kyu Park, Jack Liu, Mohammadali Ghodrat, Cao Gao, Berkin Akin, Liang-Yun Wang, Chirag Gandhi, and Dongdong Li from the Google Edge TPU team.
An Interactive, Automated 3D Reconstruction of a Fly Brain
Monday, August 5, 2019
Posted by Peter H. Li, Research Scientist and Jeremy Maitin-Shepard, Software Engineer, Connectomics at Google
The goal of
connectomics
research is to map the brain’s "wiring diagram" in order to understand how the nervous system works. A primary target of recent work is the brain of the
fruit fly
(
Drosophila melanogaster
), which is a well-established research animal in biology. Eight Nobel Prizes have been awarded for
fruit fly research
that has led to advances in molecular biology, genetics, and neuroscience. An important advantage of flies is their size:
Drosophila
brains are relatively small (one hundred thousand neurons) compared to, for example, a mouse brain (one hundred million neurons) or a human brain (one hundred billion neurons). This makes fly brains easier to study as a complete circuit.
Today, in collaboration with the Howard Hughes Medical Institute (HHMI)
Janelia Research Campus
and
Cambridge University
, we are excited to publish “
Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment
”, a new research paper that presents the automated reconstruction of an entire fruit fly brain. We are also making the full results available for anyone to
download
or to
browse online
using an interactive, 3D interface we developed called Neuroglancer.
A 40-trillion pixel fly brain reconstruction, open to anyone for
interactive viewing
. Bottom right: smaller datasets that Google AI analyzed in publications in
2016
and
2018
.
Automated Reconstruction of 40 Trillion Pixels
Our collaborators at HHMI sectioned a fly brain into thousands of ultra-thin 40-nanometer slices, imaged each slice using a transmission electron microscope (
resulting
in over forty trillion pixels of brain imagery), and then aligned the 2D images into a coherent, 3D image volume of the entire fly brain. Using thousands of
Cloud TPUs
we then applied
Flood-Filling Networks
(FFNs), which
automatically traced each individual neuron
in the fly brain.
While the algorithm generally performed well, we found performance degraded when the alignment was imperfect (image content in consecutive sections was not stable) or when occasionally there were multiple consecutive slices missing due to difficulties associated with the sectioning and imaging process. In order to compensate for these issues we combined FFNs with two new procedures. First, we estimated the slice-to-slice consistency everywhere in the 3D image and then locally stabilized the image content as the FFN traced each neuron. Second, we used a “
Segmentation-Enhanced CycleGAN
” (SECGAN) to computationally “hallucinate” missing slices in the image volume. SECGANs are a type of
generative adversarial network
specialized for image segmentation. We found that the FFN was able to trace through locations with multiple missing slices much more robustly when using the SECGAN-hallucinated image data.
Interactive Visualization of the Fly Brain with Neuroglancer
When working with 3D images that contain trillions of pixels and objects with
complicated shapes
, visualization is both essential and difficult. Inspired by Google’s history of developing
new visualization technologies
, we designed a new tool that was scalable and powerful, but also accessible to anybody with a web browser that supports WebGL.
The result is Neuroglancer
, an open-source project (
github
) that enables viewing of petabyte-scale 3D volumes, and supports many advanced features such as arbitrary-axis cross-sectional reslicing, multi-resolution meshes, and the powerful ability to develop custom analysis workflows via integration with Python. This tool has become heavily used by collaborators at the Allen Institute for Brain Science, Harvard University, HHMI, Max Planck Institute, MIT, Princeton University, and elsewhere.
A recorded demonstration of Neuroglancer. Interactive version
available here
.
Next Steps
Our collaborators at HHMI and Cambridge University have already begun using this reconstruction to accelerate their studies of learning, memory, and perception in the fly brain. However, the results described above are not yet a true
connectome
since establishing a connectome requires the identification of
synapses
. We are working closely with the
FlyEM team
at Janelia Research Campus to create a highly verified and exhaustive connectome of the fly brain using
images
acquired with “
FIB-SEM
” technology.
Acknowledgements
We would like to acknowledge core contributions from Tim Blakely, Viren Jain, Michal Januszewski, Laramie Leavitt, Larry Lindsey, Mike Tyka (Google), as well as Alex Bates, Davi Bock, Greg Jefferis, Feng Li, Mathew Nichols, Eric Perlman, Istvan Taisz, and Zhihao Zheng (Cambridge University, HHMI Janelia, Johns Hopkins University, and University of Vermont).
Robust Neural Machine Translation
Monday, July 29, 2019
Posted by Yong Cheng, Software Engineer, Google Research
In recent years,
neural machine translation
(NMT) using
Transformer models
has experienced tremendous success. Based on
deep neural networks
, NMT models are usually trained end-to-end on very large parallel corpora (input/output text pairs) in an entirely data-driven fashion and without the need to impose explicit rules of language.
Despite this huge success, NMT models can be sensitive to minor perturbations of the input, which can manifest as a variety of different errors, such as under-translation, over-translation or mistranslation. For example, given a German sentence, the state-of-the-art NMT model,
Transformer
, will yield a correct translation.
“Der Sprecher des Untersuchungsausschusses hat angekündigt, vor Gericht zu ziehen, falls sich die
geladenen
Zeugen weiterhin weigern sollten, eine Aussage zu machen.”
(Machine translation to English:
“The spokesman of the Committee of Inquiry has announced that if the witnesses summoned continue to refuse to testify, he will be brought to court.”
),
But, when we apply a subtle change to the input sentence, say from
geladenen
to the synonym
vorgeladenen
, the translation becomes very different (and in this case, incorrect):
“Der Sprecher des Untersuchungsausschusses hat angekündigt, vor Gericht zu ziehen, falls sich die
vorgeladenen
Zeugen weiterhin weigern sollten, eine Aussage zu machen.”
(Machine translation to English:
“The investigative committee has announced that he will be brought to justice if the witnesses who have been invited continue to refuse to testify.”
).
This lack of robustness in NMT models prevents many commercial systems from being applicable to tasks that cannot tolerate this level of instability. Therefore, learning robust translation models is not just desirable, but is often required in many scenarios. Yet, while the robustness of neural networks has been extensively studied in the computer vision community, only a few prior studies on learning robust NMT models can be found in literature.
In “
Robust Neural Machine Translation with Doubly Adversarial Inputs
” (to appear at
ACL 2019
), we propose an approach that uses generated adversarial examples to improve the stability of machine translation models against small perturbations in the input. We learn a robust NMT model to directly overcome adversarial examples generated with knowledge of the model and with the intent of distorting the model predictions. We show that this approach improves the performance of the NMT model on standard benchmarks.
Training a Model with AdvGen
An ideal NMT model would generate similar translations for separate inputs that exhibit small differences. The idea behind our approach is to perturb a translation model with adversarial inputs in the hope of improving the model’s robustness. It does this using an algorithm called Adversarial Generation (AdvGen
),
which generates plausible adversarial examples for perturbing the model and then feeds them back into the model for defensive training. While this method is inspired by the idea of
generative adversarial networks
(GANs), it does not rely on a discriminator network, but simply applies the adversarial example in training, effectively diversifying and extending the training set.
The first step is to perturb the model using AdvGen. We start by using Transformer to calculate the translation loss based on a source input sentence, a target input sentence and a target output sentence. Then AdvGen randomly selects some words in the source sentence, assuming a uniform distribution. Each word has an associated list of similar words, i.e., candidates that can be used for substitution, from which AdvGen selects the word that is most likely to introduce errors in Transformer output. Then, this generated adversarial sentence is fed back into Transformer, initiating the defense stage.
First, the Transformer model is applied to an input sentence (
lower left
) and, in conjunction with the target output sentence (
above right
) and target input sentence (
middle right;
beginning with the placeholder “<sos>”), the translation loss is calculated. The AdvGen function then takes the source sentence, word selection distribution, word candidates, and the translation loss as inputs to construct an adversarial source example.
During the defend stage, the adversarial sentence is fed back into the Transformer model. Again the translation loss is calculated, but this time using the adversarial source input. Using the same method as above, AdvGen uses the target input sentence, word replacement candidates, the word selection distribution calculated by the attention matrix, and the translation loss to construct an adversarial
target
example.
In the defense stage, the adversarial source example serves as input to the Transformer model, and the translation loss is calculated. AdvGen then uses the same method as above to generate an adversarial target example from the target input.
Finally, the adversarial sentence is fed back into Transformer and the robustness loss using the adversarial source example, the adversarial target input example and the target sentence is calculated. If the perturbation led to a significant loss, the loss is minimized so that when the model is confronted with similar perturbations, it will not repeat the same mistake. On the other hand, if the perturbation leads to a low loss, nothing happens, indicating that the model can already handle this perturbation.
Model Performance
We demonstrate the effectiveness of our approach by applying it to the standard Chinese-English and English-German translation benchmarks. We observed a notable improvement of 2.8 and 1.6
BLEU
points, respectively, compared to the competitive Transformer model, achieving a new state-of-the-art performance.
Comparison of Transformer model (Vaswani et al., 2017) on standard benchmarks.
We then evaluate our model on a noisy dataset, generated using a procedure similar to that described for AdvGen. We take an input clean dataset, such as that used on standard translation benchmarks, and randomly select words for similar word substitution. We find that our model exhibits improved robustness compared to other recent models.
Comparison of Transformer,
Miyao et al.
and
Cheng et al.
on artificial noisy inputs.
These results show that our method is able to overcome small perturbations in the input sentence and improve the generalization performance. It outperforms competitive translation models and achieves state-of-the-art translation performance on standard benchmarks. We hope our translation model will serve as a robust building block for improving many downstream tasks, especially when those are sensitive or intolerant to imperfect translation input.
Acknowledgements
This research was conducted by Yong Cheng, Lu Jiang and Wolfgang Macherey. Additional thanks go to our leadership Andrew Moore and Julia (Wenli) Zhu.
Google at ACL 2019
Monday, July 29, 2019
Andrew Helton, Editor, Google Research Communications
This week, Florence, Italy hosts the
2019 Annual Meeting of the Association for Computational Linguistics
(ACL 2019), the premier conference in the field of natural language understanding, covering a broad spectrum of research areas that are concerned with computational approaches to natural language.
As a leader in natural language processing and understanding, and a Diamond Level sponsor of ACL 2019, Google will be on hand to showcase the latest research on syntax, semantics, discourse, conversation, multilingual modeling, sentiment analysis, question answering, summarization, and generally building better systems using labeled and unlabeled data.
If you’re attending ACL 2019, we hope that you’ll stop by the Google booth to meet our researchers and discuss projects and opportunities at Google that go into solving interesting problems for billions of people. Our researchers will also be on hand to demo the
Natural Questions corpus
, the
Multilingual Universal Sentence Encoder
and more. You can also learn more about the Google research being presented at ACL 2019 below (Google affiliations in
blue
).
Organizing Committee
includes:
Enrique Alfonseca
Accepted Publications
A Joint Named-Entity Recognizer for Heterogeneous Tag-sets Using a Tag Hierarchy
Genady Beryozkin
,
Yoel Drori
,
Oren Gilon
,
Tzvika Hartman
,
Idan Szpektor
Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study
Chinnadhurai Sankar
, Sandeep Subramanian, Chris Pal,
Sarath Chandar
, Yoshua Bengio
Generating Logical Forms from Graph Representations of Text and Entities
Peter Shaw
,
Philip Massey
,
Angelica Chen
,
Francesco Piccinno
,
Yasemin Altun
Extracting Symptoms and their Status from Clinical Conversations
Nan Du
,
Kai Chen
,
Anjuli Kannan
,
Linh Trans
,
Yuhui Chen
,
Izhak Shafran
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
Vihan Jain
,
Gabriel Magalhaes
,
Alexander Ku
,
Ashish Vaswani
,
Eugene Le
,
Jason Baldridge
Meaning to Form: Measuring Systematicity as Information
Tiago Pimentel, Arya D. McCarthy, Damian Blasi,
Brian Roark
, Ryan Cotterell
Matching the Blanks: Distributional Similarityfor Relation Learning
Livio Baldini Soares
,
Nicholas FitzGerald
,
Jeffrey Ling
,
Tom Kwiatkowski
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Zihang Dai
,
Zhilin Yang
, Yiming Yang, Jaime Carbonell,
Quoc Le
, Ruslan Salakhutdinov
HighRES: Highlight-based Reference-less Evaluation of Summarization
Hardy Hardy,
Shashi Narayan
, Andreas Vlachos
Zero-Shot Entity Linking by Reading Entity Descriptions
Lajanugen Logeswaran,
Ming-Wei Chang
,
Kristina Toutanova
,
Kenton Lee
,
Jacob Devlin
,
Honglak Lee
Robust Neural Machine Translation with Doubly Adversarial Inputs
Yong Cheng
,
Lu Jiang
,
Wolfgang Macherey
Natural Questions: a Benchmark for Question Answering Research
Tom Kwiatkowski
,
Jennimaria Palomaki
,
Olivia Redfield
,
Michael Collins
,
Ankur Parikh
,
Chris Alberti
,
Danielle Epstein
,
Illia Polosukhin
,
Matthew Kelcey
,
Jacob Devlin
,
Kenton Lee
,
Kristina N. Toutanova
,
Llion Jones
,
Ming-Wei Chang
,
Andrew Dai
,
Jakob Uszkoreit
,
Quoc Le
,
Slav Petrov
Like a Baby: Visually Situated Neural Language Acquisition
Alexander Ororbia, Ankur Mali, Matthew Kelly,
David Reitter
What Kind of Language Is Hard to Language-Model?
Sebastian J. Mielke, Ryan Cotterell,
Kyle Gorman
,
Brian Roark
, Jason Eisner
How Multilingual is Multilingual BERT?
Telmo Pires
,
Eva Schlinger
,
Dan Garrette
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Bhuwan Dhingra,
Manaal Faruqui
,
Ankur Parikh
,
Ming-Wei Chang
,
Dipanjan Das
,
William Cohen
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Kevin Clark
,
Minh-Thang Luong
, Urvashi Khandelal, Christopher D. Manning,
Quoc V. Le
Dynamically Composing Domain-Data Selection with Clean-Data Selection by “Co-Curricular Learning" for Neural Machine Translation
Wei Wang
,
Isaac Caswell
,
Ciprian Chelba
Monotonic Infinite Lookback Attention for Simultaneous Machine Translation
Naveen Arivazhagan
,
Colin Cherry
,
Wolfgang Macherey
,
Chung-Cheng Chiu
,
Semih Yavuz
,
Ruoming Pang
,
Wei Li
,
Colin Raffel
On the Robustness of Self-Attentive Models
Yu-Lun Hsieh, Minhao Cheng,
Da-Cheng Juan
,
Wei Wei
, Wen-Lian Hsu, Cho-Jui Hsieh
Neural Decipherment via Minimum-Cost Flow: from Ugaritic to Linear B
Jiaming Luo,
Yuan Cao
, Regina Barzilay
How Large Are Lions? Inducing Distributions over Quantitative Attributes
Yanai Elazar, Abhijit Mahabal,
Deepak Ramachandran
,
Tania Bedrax-Weiss
, Dan Roth
BERT Rediscovers the Classical NLP Pipeline
Ian Tenney
,
Dipanjan Das
,
Ellie Pavlick
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas Mccoy, Roma Patel, Najoung Kim,
Ian Tenney
, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave,
Ellie Pavlick
, Samuel R. Bowman
Robust Zero-Shot Cross-Domain Slot Filling with Example Values
Darsh Shah
,
Raghav Gupta
,
Amir Fayazi
,
Dilek Hakkani-Tur
Latent Retrieval for Weakly Supervised Open Domain Question Answering
Kenton Lee
,
Ming-Wei Chang
,
Kristina Toutanova
On-device Structured and Context Partitioned Projection Networks
Sujith Ravi
,
Zornitsa Kozareva
Incorporating Priors with Feature Attribution on Text Classification
Frederick Liu
,
Besim Avci
Informative Image Captioning with External Sources of Information
Sanqiang Zhao,
Piyush Sharma
,
Tomer Levinboim
,
Radu Soricut
Reducing Word Omission Errors in Neural Machine Translation: A Contrastive Learning Approach
Zonghan Yang,
Yong Cheng
, Yang Liu, Maosong Sun
Synthetic QA Corpora Generation with Roundtrip Consistency
Chris Alberti
,
Daniel Andor
,
Emily Pitler
,
Jacob Devlin
,
Michael Collins
Unsupervised Paraphrasing without Translation
Aurko Roy
,
David Grangier
Workshops
Widening NLP 2019
Organizers include:
Diyi Yang
NLP for Conversational AI
Organizers include:
Thang-Minh Luong
,
Tania Bedrax-Weiss
The Fourth Arabic Natural Language Processing Workshop
Organizers include:
Imed Zitouni
The Third Workshop on Abusive Language Online
Organizers include:
Zeerak Waseem
TyP-NLP, Typology for Polyglot NLP
Organizers include:
Manaal Faruqui
Gender Bias in Natural Language Processing
Organizers include:
Kellie Webster
Tutorials
Wikipedia as a Resource for Text Analysis and Retrieval
Organizer:
Marius Pasca
Learning Better Simulation Methods for Partial Differential Equations
Tuesday, July 23, 2019
Posted by Stephan Hoyer, Software Engineer, Google Research
The world’s fastest supercomputers were designed for modeling physical phenomena, yet they still are not fast enough to robustly predict the
impacts of climate change
, to design controls for
airplanes
based on airflow or to accurately
simulate a fusion reactor
. All of these phenomena are modeled by
partial differential equations
(PDEs), the class of equations that describe everything smooth and continuous in the physical world, and the most common class of simulation problems in science and engineering. To solve these equations, we need faster simulations, but in recent years,
Moore’s law has been slowing
. At the same time, we’ve seen huge breakthroughs in machine learning (ML) along with
faster hardware
optimized for it. What does this new paradigm offer for scientific computing?
In “
Learning Data Driven Discretizations for Partial Differential Equations
”, published in
Proceedings of the National Academy of Sciences
, we explore a potential path for how ML can offer continued improvements in high-performance computing, both for solving PDEs and, more broadly, for solving hard computational problems in every area of science.
For most real-world problems,
closed-form solutions
to PDEs don’t exist. Instead, one must find discrete equations (“
discretizations”
) that a computer can solve to approximate the continuous PDE. Typical approaches to solve PDEs represent equations on a grid, e.g., using
finite differences
. To achieve convergence, the mesh spacing of the grid needs to be smaller than the smallest feature size of the solutions. This often isn’t feasible because of an unfortunate scaling law: achieving 10x higher resolution requires 10,000x more compute, because the grid must be scaled in four dimensions—three spatial dimensions and time. Instead, in our paper we show that ML can be used to learn better representations for PDEs on coarser grids.
Satellite photo of a hurricane, at both full resolution and simulated resolution in a
state-of-the-art weather model
. Cumulus clouds (e.g., in the red circle) are responsible for heavy rainfall, but in the weather model the details are entirely blurred out. Instead, models rely on
crude approximations
for sub-grid physics, a
key source of uncertainty
in climate models. Image credit:
NOAA
The challenge is to retain the accuracy of high-resolution simulations while still using the coarsest grid possible. In our work we’re able to improve upon
existing schemes
by replacing heuristics based on deep human insight (e.g., “solutions to a PDE should always be smooth away from discontinuities”) with optimized rules based on machine learning. The rules our ML models recover are complex, and we don’t entirely understand them, but they incorporate sophisticated physical principles like the idea of “upwinding”—to accurately model what’s coming towards you in a fluid flow, you should look upstream in the direction the wind is coming from. An example of our results on a simple model of fluid dynamics are shown below:
Simulations of
Burgers’ equation
, a model for shock waves in fluids, solved with either a standard finite volume method (left) or our neural network based method (right). The orange squares represent simulations with each method on low resolution grids. These points are fed back into the model at each time step, which then predicts how they should change. Blue lines show the exact simulations used for training. The neural network solution is much better, even on a 4x coarser grid, as indicated by the orange squares smoothly tracing the blue line.
Our research also illustrates a broader lesson about how to effectively combine machine learning and physics. Rather than attempting to learn physics from scratch, we combined neural networks with components from traditional simulation methods, including the known form of the equations we’re solving and
finite volume methods
. This means that laws such as
conservation of momentum
are exactly satisfied, by construction, and allows our machine learning models to focus on what they do best, learning optimal rules for interpolation in complex, high-dimensional spaces.
Next Steps
We are focused on scaling up the techniques outlined in our paper to solve larger scale simulation problems with real-world impacts, such as weather and climate prediction. We’re excited about the broad potential of
blending machine learning
into the complex algorithms of scientific computing.
Acknowledgments
Thanks to co-authors Yohai Bar-Sinari, Jason Hickey and Michael Brenner; and Google collaborators Peyman Milanfar, Pascal Getreur, Ignacio Garcia Dorado, Dmitrii Kochkov, Jiawei Zhuang and Anton Geraschenko.
Building SMILY, a Human-Centric, Similar-Image Search Tool for Pathology
Friday, July 19, 2019
Posted by Narayan Hegde, Software Engineer, Google Health and Carrie J. Cai, Research Scientist, Google Research
Advances in machine learning (ML) have shown great promise for assisting in the work of healthcare professionals, such as aiding the detection of
diabetic eye disease
and
metastatic breast cancer
. Though high-performing algorithms are necessary to gain the trust and adoption of clinicians, they are not always sufficient—
what
information is presented to doctors and
how
doctors interact with that information can be crucial determinants in the utility that ML technology ultimately has for users.
The medical specialty of
anatomic pathology
, which is the gold standard for the diagnosis of cancer and many other diseases through microscopic analysis of tissue samples, can greatly benefit from applications of ML. Though diagnosis through pathology is traditionally done on physical microscopes, there has been a growing adoption of “digital pathology,” where high-resolution images of pathology samples can be examined on a computer. With this movement comes the potential to much more easily look up information, as is needed when pathologists tackle the diagnosis of difficult cases or rare diseases, when “general” pathologists approach specialist cases, and when trainee pathologists are learning. In these situations, a common question arises, “What is this feature that I’m seeing?” The traditional solution is for doctors to ask colleagues, or to laboriously browse reference textbooks or online resources, hoping to find an image with similar visual characteristics. The general computer vision solution to problems like this is termed
content-based image retrieval
(CBIR), one example of which is the “
reverse image search
” feature in
Google Images
, in which users can search for similar images by using another image as input.
Today, we are excited to share two research papers describing further progress in human-computer interaction research for
similar image search
in medicine. In “
Similar Image Search for Histopathology: SMILY
” published in
Nature Partner Journal (npj) Digital Medicine
, we report on our ML-based tool for reverse image search for pathology. In our second paper,
“
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
”
(preprint available
here
), which received an
honorable mention
at the
2019 ACM CHI Conference on Human Factors in Computing Systems
, we explored different modes of refinement for image-based search, and evaluated their effects on doctor interaction with SMILY.
SMILY Design
The first step in developing SMILY was to apply a deep learning model, trained using 5 billion natural, non-pathology images (e.g., dogs, trees, man-made objects, etc.), to compress images into a “summary” numerical vector, called an
embedding
. The network learned during the training process to distinguish similar images from dissimilar ones by computing and comparing their embeddings. This model is then used to create a database of image patches and their associated embeddings using a corpus of de-identified slides from
The Cancer Genome Atlas
. When a query image patch is selected in the SMILY tool, the query patch’s embedding is similarly computed and compared with the database to retrieve the image patches with the most similar embeddings.
Schematic of the steps in building the SMILY database and the process by which input image patches are used to perform the similar image search.
The tool allows a user to select a region of interest, and obtain visually-similar matches. We tested SMILY’s ability to retrieve images along a pre-specified axis of similarity (e.g. histologic feature or tumor grade), using images of tissue from the breast, colon, and prostate (3 of the most common cancer sites). We found that SMILY demonstrated promising results despite not being trained specifically on pathology images or using any labeled examples of histologic features or tumor grades.
Example of selecting a small region in a slide and using SMILY to retrieve similar images. SMILY efficiently searches a database of billions of cropped images in a few seconds. Because pathology images can be viewed at different magnifications (zoom levels), SMILY automatically searches images at the same magnification as the input image.
Second example of using SMILY, this time searching for a lobular carcinoma, a specific subtype of breast cancer.
Refinement tools for SMILY
However, a problem emerged when we observed how pathologists interacted with SMILY. Specifically, users were trying to answer the nebulous question of “What
looks similar
to this image?” so that they could learn from past cases containing similar images. Yet, there was no way for the tool to understand the intent of the search: Was the user trying to find images that have a similar histologic feature, glandular morphology, overall architecture, or something else? In other words, users needed the ability to guide and refine the search results on a case-by-case basis in order to actually find what they were looking for. Furthermore, we observed that this need for iterative search refinement was rooted in how doctors often perform “
iterative diagnosis
”—by generating hypotheses, collecting data to test these hypotheses, exploring alternative hypotheses, and revisiting or retesting previous hypotheses in an iterative fashion. It became clear that, for SMILY to meet real user needs, it would need to support a different approach to user interaction.
Through careful human-centered research described in our
second paper
, we designed and augmented SMILY with a suite of interactive refinement tools that enable end-users to express what similarity means on-the-fly: 1)
refine-by-region
allows pathologists to crop a region of interest within the image, limiting the search to just that region; 2)
refine-by-example
gives users the ability to pick a subset of the search results and retrieve more results like those; and 3)
refine-by-concept
sliders can be used to specify that more or less of a clinical concept be present in the search results (e.g., fused glands). Rather than requiring that these concepts be built into the machine learning model, we instead developed a method that enables end-users to create new concepts post-hoc, customizing the search algorithm towards concepts they find important for each specific use case. This enables new explorations via post-hoc tools after a machine learning model has already been trained, without needing to re-train the original model for each concept or application of interest.
Through our user study with pathologists, we found that the tool-based SMILY not only increased the clinical usefulness of search results, but also significantly increased users’ trust and likelihood of adoption, compared to a conventional version of SMILY without these tools. Interestingly, these refinement tools appeared to have supported pathologists’ decision-making process in ways
beyond
simply performing better on similarity searches. For example, pathologists used the observed changes to their results from iterative searches as a means of progressively tracking the likelihood of a hypothesis. When search results were surprising, many re-purposed the tools to test and understand the underlying algorithm, for example, by cropping out regions they thought were interfering with the search or by adjusting the concept sliders to increase the presence of concepts they suspected were being ignored. Beyond being passive recipients of ML results, doctors were empowered with the agency to actively test hypotheses and apply their expert domain knowledge, while simultaneously leveraging the benefits of automation.
With these interactive tools enabling users to tailor each search experience to their desired intent, we are excited for SMILY’s potential to assist with searching large databases of digitized pathology images. One potential application of this technology is to index textbooks of pathology images with descriptive captions, and enable medical students or pathologists in training to search these textbooks using visual search, speeding up the educational process. Another application is for cancer researchers interested in studying the correlation of tumor morphologies with patient outcomes, to accelerate the search for similar cases. Finally, pathologists may be able to leverage tools like SMILY to locate all occurrences of a feature (e.g. signs of active cell division, or mitosis) in the same patient’s tissue sample to better understand the severity of the disease to inform cancer therapy decisions. Importantly, our findings add to the body of evidence that sophisticated machine learning algorithms need to be paired with human-centered design and interactive tooling in order to be most useful.
Acknowledgements
This work would not have been possible without Jason D. Hipp, Yun Liu, Emily Reif, Daniel Smilkov, Michael Terry, Craig H. Mermel, Martin C. Stumpe and members of
Google Health
and
PAIR
. Preprints of the two papers are available
here
and
here
.
Labels
2018
accessibility
ACL
ACM
Acoustic Modeling
Adaptive Data Analysis
ads
adsense
adwords
Africa
AI
Algorithms
Android
Android Wear
API
App Engine
App Inventor
April Fools
Art
Audio
Augmented Reality
Australia
Automatic Speech Recognition
AutoML
Awards
BigQuery
Cantonese
Chemistry
China
Chrome
Cloud Computing
Collaboration
Compression
Computational Imaging
Computational Photography
Computer Science
Computer Vision
conference
conferences
Conservation
correlate
Course Builder
crowd-sourcing
CVPR
Data Center
Data Discovery
data science
datasets
Deep Learning
DeepDream
DeepMind
distributed systems
Diversity
Earth Engine
economics
Education
Electronic Commerce and Algorithms
electronics
EMEA
EMNLP
Encryption
entities
Entity Salience
Environment
Europe
Exacycle
Expander
Faculty Institute
Faculty Summit
Flu Trends
Fusion Tables
gamification
Gboard
Gmail
Google Accelerated Science
Google Books
Google Brain
Google Cloud Platform
Google Docs
Google Drive
Google Genomics
Google Maps
Google Photos
Google Play Apps
Google Science Fair
Google Sheets
Google Translate
Google Trips
Google Voice Search
Google+
Government
grants
Graph
Graph Mining
Hardware
HCI
Health
High Dynamic Range Imaging
ICCV
ICLR
ICML
ICSE
Image Annotation
Image Classification
Image Processing
Inbox
India
Information Retrieval
internationalization
Internet of Things
Interspeech
IPython
Journalism
jsm
jsm2011
K-12
Kaggle
KDD
Keyboard Input
Klingon
Korean
Labs
Linear Optimization
localization
Low-Light Photography
Machine Hearing
Machine Intelligence
Machine Learning
Machine Perception
Machine Translation
Magenta
MapReduce
market algorithms
Market Research
Mixed Reality
ML
ML Fairness
MOOC
Moore's Law
Multimodal Learning
NAACL
Natural Language Processing
Natural Language Understanding
Network Management
Networks
Neural Networks
NeurIPS
Nexus
Ngram
NIPS
NLP
On-device Learning
open source
operating systems
Optical Character Recognition
optimization
osdi
osdi10
patents
Peer Review
ph.d. fellowship
PhD Fellowship
PhotoScan
Physics
PiLab
Pixel
Policy
Professional Development
Proposals
Public Data Explorer
publication
Publications
Quantum AI
Quantum Computing
Reinforcement Learning
renewable energy
Research
Research Awards
resource optimization
Robotics
schema.org
Search
search ads
Security and Privacy
Semantic Models
Semi-supervised Learning
SIGCOMM
SIGMOD
Site Reliability Engineering
Social Networks
Software
Sound Search
Speech
Speech Recognition
statistics
Structured Data
Style Transfer
Supervised Learning
Systems
TensorBoard
TensorFlow
TPU
Translate
trends
TTS
TV
UI
University Relations
UNIX
Unsupervised Learning
User Experience
video
Video Analysis
Virtual Reality
Vision Research
Visiting Faculty
Visualization
VLDB
Voice Search
Wiki
wikipedia
WWW
Year in Review
YouTube
Archive
2019
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2018
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2017
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Jul
May
Apr
Mar
Feb
2007
Oct
Sep
Aug
Jul
Jun
Feb
2006
Dec
Nov
Sep
Aug
Jul
Jun
Apr
Mar
Feb
Feed
Follow @googleai
Give us feedback in our
Product Forums
.
