Google AI Blog: February 2020

Ultra-High Resolution Image Analysis with Mesh-TensorFlow

Friday, February 28, 2020

Posted by Le Hou and Youlong Cheng, Software Engineers, Google Researchdatamodelconvolutional neural networkcomputed tomography8High Resolution Medical Image Analysis with Spatial PartitioningMayo ClinicMesh-TensorFlowhalo exchangeU-Netopen-sourcedData and Model Parallelism with Mesh-TensorFlowtensorsxy

Spatial partitioning of ultra-high resolution images, in this case, a 3D CT scan.

Spatial Partitioning with Halo Exchangeconvolution operationhalo exchange

Halo exchange ensures that cross-partition convolutions handle image segment edges correctly.

Proof of Concept - Segmentation of Liver Tumor CT ScansLiTS benchmarkSørensen–Dice coefficient3xyz

Higher resolution data yields better segmentation accuracy.

Conclusion8with the released codeAcknowledgmentsWe thank our collaborators Panagiotis Korfiatis, Ph.D., and Daniel Blezek, Ph.D., from Mayo Clinic for providing the initial 3D U-net model and training data. Thank you Greg Mikels for the POC work with Mayo Clinic. Special thanks to all the co-authors of the paper especially Noam Shazeer.

Open Images V6 — Now Featuring Localized Narratives

Wednesday, February 26, 2020

Posted by Jordi Pont-Tuset, Research Scientist, Google ResearchOpen Imagesdeep convolutional neural networksintroduction of version 5object detectioninstance segmentationvisual relationship detection

Annotation modalities in Open Images V5: image-level labels, bounding boxes, instance segmentations, and visual relationships. Image sources: 1969 Camaro RS/SS by D. Miller, the house by anita kluska, Cat Cafe Shinjuku calico by Ari Helminen, and Radiofiera - Villa Cordellina Lombardi, Montecchio Maggiore (VI) - agosto 2010 by Andrea Sartorati. All images used under CC BY 2.0 license.

Open Images V6localized narrativesCOCO dataset

Sample of localized narratives. Image source: Spring is here:-) by Kasia.

Localized Narrativessome previous works

Different levels of grounding between image content and captioning. Left to Right: Caption to whole image (COCO); nouns to boxes (Flickr30k Entities); each word to a mouse trace segment (localized narratives). Image sources: COCO, Flickr30k Entities, and Sapa, Vietnam by Rama.

Localized narratives

Alignment of manual and automatic transcriptions. Icons based on an original design from Freepik.

Mouse trace segments corresponding to the words below the images. Image sources: Via Guglielmo Marconi, Positano - Hotel Le Agavi - boat by Elliott Brown, air frame by vivek jena, and CL P1050512 by Virginia State Parks.

New Visual Relationships, Human Actions, and Image-Level Annotations

Image sources: IMG_5678.jpg by James Buck, DSC_0494 by Quentin Meulepas, and DSC06464 by sally9258.

Image sources: _DSCs1341 (2) by Boo Ph, and Richard Wagner Spiele 2015 by Johannes Gärtner.

Conclusion

Enhancing the Research Community’s Access to Street View Panoramas for Language Grounding Tasks

Tuesday, February 25, 2020

Posted by Harsh Mehta, Software Engineer and Jason Baldridge, Research Scientist, Google Researchnatural language processingcomputer visionincluding robotsGo forward and turn left after the red fire hydrant by the train tracks. Then go three blocks and stop in front of the building with a row of flags over its entrance.train tracksred fire hydrantrow of flagsblockbyover, go forwardturn leftStreet ViewTouchdownvision-and-language navigationspatial description resolutionRetouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street ViewStreetLearnopen source TensorFlow implementations for the Touchdown tasksVALANGrounded Language Understanding Tasks

Example animation of a person following Touchdown instructions: “Orient yourself so that the umbrellas are to the right. Go straight and take a right at the first intersection. At the next intersection there should be an old-fashioned store to the left. There is also a dinosaur mural to the right. Touchdown is on the back of the dinosaur.”

Room-to-Roombaseline models in TouchdownGoogle Maps/Google Earth Terms of ServiceStreetLearn: A Dataset of Approved Panoramas for Research UseDeepMindStreetLearnthe StreetNav task suiteRoom-to-Room;transient objects

Touchdown instruction: “Two parked bicycles, and a discarded couch, all on the left. Walk just past this couch, and stop before you pass another parked bicycle. This bike will be white and red, with a white seat. Touchdown is sitting on top of the bike seat.” Other panoramas from the same location taken at other times would be highly unlikely to contain these exact items in the exact same positions. For a concrete example, see the current imagery available for this location in Street View, which contains very different transient objects.

A panorama one step farther down the street from the previous scene.

granularity

Adding Touchdown Panoramas to StreetLearn and Verifying Model BaselinesRetouchdownVALANChen et al. (2019)SDTW metricaccuracy@pxnsmall differences in models and processing

Performance comparison between Chen et al. (2019) using the original panoramas (in blue) and our reimplementation using the panoramas available in StreetLearn (in red). Top: VLN results for task completion, shortest path distance and success weighted by Dynamic Time Warping (SDTW). Bottom: SDR results for the accuracy@npx metrics.

Obtaining the DataStreetLearn interest formTouchdown github repositoryAcknowledgementsThe core team includes Yoav Artzi, Eugene Ie, and Piotr Mirowski. We would like to thank Howard Chen for his help with reproducing the Touchdown results, Larry Lansing, Valts Blukis and Vihan Jain for their help with the code and open-sourcing, and the Language team in Google Research, especially Radu Soricut, for the insightful comments that contributed to this work. Many thanks also to the Google Maps and Google Street View teams for their support in accessing and releasing the data, and to the Data Compute team for reviewing the panoramas.

Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer

Monday, February 24, 2020

Posted by Adam Roberts, Staff Software Engineer and Colin Raffel, Senior Research Scientist, Google Researchnatural language processingGPTULMFiTELMoBERTXLNetRoBERTaALBERTReformerMT-DNNExploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerColossal Clean Crawled Corpuscodepre-trained modelsColab NotebookA Shared Text-To-Text Frameworkany

Diagram of our text-to-text framework. Every task we consider uses text as input to the model, which is trained to generate some target text. This allows us to use the same model, loss function, and hyperparameters across our diverse set of tasks including translation (green), linguistic acceptability (red), sentence similarity (yellow), and document summarization (blue). It also provides a standard testbed for the methods included in our empirical survey.

A Large Pre-training Dataset (C4)WikipediaCommon Crawl two orders of magnitude larger than WikipediaTensorFlow DatasetsA Systematic Study of Transfer Learning Methodology

model architectures, where we found that encoder-decoder models generally outperformed "decoder-only" language models;

pre-training objectives, where we confirmed that fill-in-the-blank-style denoising objectives (where the model is trained to recover missing words in the input) worked best and that the most important factor was the computational cost;

unlabeled datasets, where we showed that training on in-domain data can be beneficial but that pre-training on smaller datasets can lead to detrimental overfitting;

training strategies, where we found that multitask learning could be close to competitive with a pre-train-then-fine-tune approach but requires carefully choosing how often the model is trained on each task;

and scale, where we compare scaling up the model size, the training time, and the number of ensembled models to determine how to make the best use of fixed compute power.

Insights + Scale = State-of-the-ArtGoogle Cloud TPU acceleratorsGLUESuperGLUESQuADCNN/Daily MailExtensionsClosed-Book Question AnsweringHurricane ConnieOn what date did Hurricane Connie occur?Colabpaperwithout access to any external knowledgeopen-domain question answering

During pre-training, T5 learns to fill in dropped-out spans of text (denoted by <M>) from documents in C4. To apply T5 to closed-book question answer, we fine-tuned it to answer questions without inputting any additional information or context. This forces T5 to answer questions based on “knowledge” that it internalized during pre-training.

TriviaQAWebQuestionsNatural Questions

Fill-in-the-Blank Text GenerationGPT-2Talk To TransformerAI Dungeonsized fill-in-the-blankI like to eat peanut butter and _4_ sandwiches,I love peanut butter and _N_ sandwiches,.fill{color:#4184F3;font-style:italic}.monospace{font-family:'Roboto Mono',monospace}.tabs{width:100%!important;margin:0!important;padding:0!important;list-style:none;position:relative;height:200px!important;margin-bottom:24px!important}.tabs li{display:inline-block;margin:0!important;padding:0!important;line-height:1!important}.tabs input[type="radio"]{position:absolute;top:0;left:0;opacity:0}.tabs label{display:block;padding:8px;width:50px;cursor:pointer;border-bottom:2px solid white;text-align:center;transition:.2s all ease-in-out}.tabs [id^="tab"]:checked+label{border-bottom:2px solid #4184F3;background:#e3edfd;color:#4184F3;font-weight:bold}.tabs label:hover{color:#4184F3;border-bottom:2px solid #4184F3;xbackground:#e6e9ff}.tabs .tab-content{z-index:2;display:none;overflow:auto;width:100%;height:160px;margin-top:0;padding:8px;position:absolute;left:0;line-height:1.5;border:1px solid #4184F3;border-top-color:#e6e9ff;border-left-color:#e6e9ff}.tabs .tab-content>p{margin-top:0}.tabs [id^="tab"]:checked ~ .tab-content{display:block;box-sizing:border-box;}@media only screen and (max-width:680px){.tabs .tab-content{margin-top:40px!important}.tabs{height:240px!important}}@media only screen and (max-width:500px){#content7,#content6{margin-top:0!important}}@media only screen and (max-width:460px){#content5{margin-top:0!important}}@media only screen and (max-width:400px){#content7,#content6,#content5{margin-top:0!important}}@media only screen and (max-width:360px){#content4{margin-top:0!important}}

N=1

I love peanut butter and jelly sandwiches.

N=2

I love peanut butter and jelly on my sandwiches.

N=4

I love peanut butter and jelly, which is what makes good sandwiches.

N=8

I love peanut butter and jelly, Yum! You can’t beat peanut butter and jelly sandwiches.

N=16

I love peanut butter and bread. Thanks!! This looks delicious. I love all types of peanut butter, but especially peanut butter/jam sandwiches.

N=32

I love peanut butter and banana sandwiches and the Peanut Butter Chocolate Chip Cookie Bites are now very easy to prepare. The best part is that the PB&J Cookie Bites are the perfect size for kid lunch boxes and everyone will enjoy them. Kids love these sandwiches.

N=64

I love peanut butter and jelly! It's like an incredible childhood love now that I'm in my 30s. I'm obsessed with peanut butter and jelly sandwiches. I have made this recipe many times with great success. I'm guessing you could probably find that it's from a 1950s cookbook. I haven't seen it listed in a current one but it is in a 1930 version (they seem to have retired it). Hope you enjoy it ;) I hope you'll post what you do with your sandwiches.

N=512

I love peanut butter and banana sandwiches. Peanut butter and bananas were the original sandwich spread (also known as PB&J or Peanut Butter and Jelly), so they are probably my favorite. When I was a kid, I knew what peanut butter and bananas tasted like, but I didn't really think of them as one flavor. I did recognize PB & J's as just a sandwich spread, and not really two separate flavours. When I was in graduate school researching food allergies and intolerances, PB&B is one of the main themes that came up. PB&B was and still is the favorite sandwich spread on my school cafeteria tray, and it's still on my personal board in the kitchen. I've seen PB&B on more than one occasion over the years. I can always find one of the other types of peanut butter spread at the school cafeteria and I find it's the perfect spread for PB&B. You have several options for making your own peanut butter (I love this recipe from Cookie and Kate) or buying it at stores like Trader Joes, Publix, and Safeway. The flavor of PB&B varies, but I've found that white chocolate is my favorite to use. This simple sandwich is perfect for lunch or an afternoon snack. When you're on the go, you can use a store bought product, but it just wouldn't taste as good. These peanut butter n' banana sandwich spreads (as well as other peanut butter spreads) are great for on the go eating, but they can also be refrigerated for a few hours and then served at your next party. Preheat oven to 350 degrees. Using a kitchen timer, or using a microwave, heat butter in a saucepan and melt over low heat. Add vanilla extract and milk and heat until nearly boiling. Add peanut butter, bananas, sugar, salt, and pepper, mixing well to combine all ingredients. Spread peanut butter mixture on both sides of your bread. Top with banana slices and a slice of cheese. Toast the bread until both sides are golden brown. Remove from toast and sprinkle a little more sugar and peanut butter on the top. Enjoy while the bread is warm. Store in an airtight container up to one day. Assemble peanut butter and banana sandwich spread by spreading the peanut butter mixture on each slice of bread. Add a banana slice on top and then a PB & J sandwich. Enjoy while the bread is still warm. P.S. You might also like these peanut butter and jelly sandwiches.

Conclusionfindingscodepre-trained modelsColab NotebookAcknowledgementsThis work has been a collaborative effort involving Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, Karishma Malkan, Noah Fiedel, and Monica Dinculescu.

Announcing the 2019 Google Faculty Research Award Recipients

Monday, February 24, 2020

Posted by Maggie Johnson, VP and Negar Saei, Program Manager, Google ResearchGoogle Faculty Research Awardsresearch philosophyMachine LearningSystemsHuman Computer Interactionwell-deserving recipientsour website

Setting Fairness Goals with the TensorFlow Constrained Optimization Library

Friday, February 21, 2020

Posted by Andrew Zaldivar, Responsible AI Advocate, Google Research, on behalf of the TFCO Teamloss functionfairnessTensorFlow Constrained Optimizationprecisionstrue positive ratesrecall ratesequalized oddspredictive parityHow Does TFCO Relate to Our AI Principles?AI PrinciplesWhat Are the Goals?Hardt et al.classifierpredictiondatasetlabelobjective functionalsofairness constraints

Illustration of a binary classification dataset with two protected groups: blue and orange. For ease of visualization, rather than plotting each individual data point, the densities are represented as ovals. The positive and negative signs denote the labels. The decision boundary drawn as a black dashed line separating positive predictions (regions above the line) and negative (regions below the line) labels, chosen to maximize accuracy.

unfair“equal opportunity” principle

Here the decision boundary is chosen to maximize the accuracy, subject to an equal opportunity (or true positive rate) constraint.

How Do I Know What Constraints To Set?contradictory constraintsfalse positive ratesin addition totrue positive ratesequalized odds

Here the decision boundary is chosen to maximize the accuracy, subject to both the true positive rate and false positive rate constraints.

eitherorCan It Fail?decreasingThe Curse of Small Sample Sizesstochastic gradientsWiki toxicity example

Comparison of unconstrained and constrained CNN models for classifying toxic comments on Wiki Talk pages.

Intersectionality – The Challenge of Fine Grained GroupsCelebA examplefalse positive rate

Comparison of unconstrained and constrained smiling detection classifier across age group categories.

Under the HoodcountsALT’19JMLR’19AAAI’20NeurIPS’19aICML’19TFCOtoxicity classificationsmile detectionAcknowledgementsThis work was a collaborative effort by the authors of TFCO and associated research papers, including Andrew Cotter, Maya R. Gupta, Heinrich Jiang, Harikrishna Narasimhan, Taman Narayan, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You.

Generating Diverse Synthetic Medical Image Data for Training Machine Learning Models

Wednesday, February 19, 2020

Posted by Timo Kohlberger and Yuan Liu, Software Engineers, Google HealthprogressGenerating Diverse Images of Skin ConditionsDermGAN: Synthetic Generation of Clinical Skin Images with PathologyMachine Learning for HealthNeurIPS 2019dermatology imagesFitzpatrick skin typesrelative few cases at the “boundaries”pix2pixgenerative adversarial networkU-Netcheckerboard artifacts.

Overview of the generator component of DermGAN. The model takes an RGB semantic map (red box) annotated with the skin condition's size and location (smaller orange rectangle), and outputs a realistic skin image. Colored boxes represent various neural network layers, such as convolutions and ReLU; the skip connections resemble the U-Net and enable information to be propagated at the appropriate scales.

The top row shows generated synthetic examples and the bottom row illustrates real images of basal cell carcinoma (left) and melanocytic nevus (right). More examples can be found in the paper.

DermGAN can be used to generate skin images (all with melanocytic nevus in this case) with different background skin types (top, by changing the input skin color) and different-sized lesions (bottom, by changing the input lesion size). As the input skin color changes, the lesion changes appearance to match what the lesion would look like on different skin types.

Generating Pathology Images with Different Labels Across Diverse Scannersmetastatic breast cancer detection algorithmsWhole-slide image focus quality: Automatic assessment and impact on AI cancer detectionJournal of Pathology Informatics

A sequence of images showing step-wise out-of-focus image generation. Images are shown in grayscale to accentuate the difference between steps. First, an in-focus image is collected (a) and a bokeh effect is added to produce a blurry image (b). Next, sensor noise is added to simulate a real image sensor (c), and finally JPEG compression is added to simulate the sharp edges introduced by post-acquisition software processing (d). A real out-of-focus image is shown for comparison (e).

modeling each step is essential

An example of a particularly interesting out-of-focus pattern across a biological tissue slice. Areas in blue were recognized by the model to be in-focus, whereas areas highlighted in yellow, orange, or red were more out of focus. The gradation in focus here (represented by concentric circles: a red/orange out-of-focus center surrounded by green/cyan mildly out-of-focus, and then a blue in-focus ring) was caused by a hard “stone” in the center that lifted the surrounding biological tissue.

Implications and Future Outlooktrainingvalidationonlyinverse probability weightingour work on ML for chest X-raysAcknowledgementsThese projects involved the efforts of multidisciplinary teams of software engineers, researchers, clinicians and cross functional contributors. Key contributors to these projects include Timo Kohlberger, Yun Liu, Melissa Moran, Po-Hsuan Cameron Chen, Trissia Brown, Jason Hipp, Craig Mermel, Martin Stumpe, Amirata Ghorbani, Vivek Natarajan, David Coz, and Yuan Liu. The authors would also like to acknowledge Daniel Fenner, Samuel Yang, Susan Huang, Kimberly Kanada, Greg Corrado and Erica Brand for their advice, members of the Google Health dermatology and pathology teams for their support, and Ashwin Kakarla and Shivamohan Reddy Garlapati for their team for image labeling.

AutoFlip: An Open Source Framework for Intelligent Video Reframing

Thursday, February 13, 2020

Posted by Nathan Frey, Senior Software Engineer, Google Research, Los Angeles and Zheng Sun, Senior Software Engineer, Google Research, Mountain Viewincreasing number of usersAutoFlipMediaPipe

Left: Original video (16:9). Middle: Reframed using a standard central crop (9:16). Right: Reframed with AutoFlip (9:16). By detecting the subjects of interest, AutoFlip is able to avoid cropping off important visual content.

AutoFlip Overview

Shot (Scene) DetectionVideo Content Analysisfaceobject detection modelsTensorFlow Lite

Left: People detection on sports footage. Right: Two face boxes (‘core’ and ‘all’ face landmarks). In narrow portrait crop cases, often only the core landmark box can fit.

ReframingstationarypanningtrackingstationarypantrackingEuclidean-normlow-degree polynomial

Top: Camera paths resulting from following the bounding boxes from frame-to-frame. Bottom: Final smoothed camera paths generated using Euclidean-norm path formation. Left: Scene in which objects are moving around, requiring a tracking camera path. Right: Scene where objects stay close to the same position; a stationary camera covers the content for the full duration of the scene.

best-effortrequiredletterbox effect

AutoFlip Use Cases

What’s Next?deep uncropAcknowledgmentsWe would like to thank our colleagues who contributed to Autoflip, Alexander Panagopoulos, Jenny Jin, Brian Mulford, Yuan Zhang, Alex Chen, Xue Yang, Mickey Wang, Justin Parra, Hartwig Adam, Jingbin Wang, and Weilong Yang; MediaPipe team who helped with open sourcing, Jiuqiang Tang, Tyler Mullen, Mogan Shieh, Ming Guang Yong, and Chuo-Ling Chang.