Google AI Blog: February 2020

Enhancing the Research Community’s Access to Street View Panoramas for Language Grounding Tasks

Tuesday, February 25, 2020

Posted by Harsh Mehta, Software Engineer and Jason Baldridge, Research Scientist, Google Researchnatural language processingcomputer visionincluding robotsGo forward and turn left after the red fire hydrant by the train tracks. Then go three blocks and stop in front of the building with a row of flags over its entrance.train tracksred fire hydrantrow of flagsblockbyover, go forwardturn leftStreet ViewTouchdownvision-and-language navigationspatial description resolutionRetouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street ViewStreetLearnopen source TensorFlow implementations for the Touchdown tasksVALANGrounded Language Understanding Tasks

Example animation of a person following Touchdown instructions: “Orient yourself so that the umbrellas are to the right. Go straight and take a right at the first intersection. At the next intersection there should be an old-fashioned store to the left. There is also a dinosaur mural to the right. Touchdown is on the back of the dinosaur.”

Room-to-Roombaseline models in TouchdownGoogle Maps/Google Earth Terms of ServiceStreetLearn: A Dataset of Approved Panoramas for Research UseDeepMindStreetLearnthe StreetNav task suiteRoom-to-Room;transient objects

Touchdown instruction: “Two parked bicycles, and a discarded couch, all on the left. Walk just past this couch, and stop before you pass another parked bicycle. This bike will be white and red, with a white seat. Touchdown is sitting on top of the bike seat.” Other panoramas from the same location taken at other times would be highly unlikely to contain these exact items in the exact same positions. For a concrete example, see the current imagery available for this location in Street View, which contains very different transient objects.

A panorama one step farther down the street from the previous scene.

granularity

Adding Touchdown Panoramas to StreetLearn and Verifying Model BaselinesRetouchdownVALANChen et al. (2019)SDTW metricaccuracy@pxnsmall differences in models and processing

Performance comparison between Chen et al. (2019) using the original panoramas (in blue) and our reimplementation using the panoramas available in StreetLearn (in red). Top: VLN results for task completion, shortest path distance and success weighted by Dynamic Time Warping (SDTW). Bottom: SDR results for the accuracy@npx metrics.

Obtaining the DataStreetLearn interest formTouchdown github repositoryAcknowledgementsThe core team includes Yoav Artzi, Eugene Ie, and Piotr Mirowski. We would like to thank Howard Chen for his help with reproducing the Touchdown results, Larry Lansing, Valts Blukis and Vihan Jain for their help with the code and open-sourcing, and the Language team in Google Research, especially Radu Soricut, for the insightful comments that contributed to this work. Many thanks also to the Google Maps and Google Street View teams for their support in accessing and releasing the data, and to the Data Compute team for reviewing the panoramas.

Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer

Monday, February 24, 2020

Posted by Adam Roberts, Staff Software Engineer and Colin Raffel, Senior Research Scientist, Google Researchnatural language processingGPTULMFiTELMoBERTXLNetRoBERTaALBERTReformerMT-DNNExploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerColossal Clean Crawled Corpuscodepre-trained modelsColab NotebookA Shared Text-To-Text Frameworkany

Diagram of our text-to-text framework. Every task we consider uses text as input to the model, which is trained to generate some target text. This allows us to use the same model, loss function, and hyperparameters across our diverse set of tasks including translation (green), linguistic acceptability (red), sentence similarity (yellow), and document summarization (blue). It also provides a standard testbed for the methods included in our empirical survey.

A Large Pre-training Dataset (C4)WikipediaCommon Crawl two orders of magnitude larger than WikipediaTensorFlow DatasetsA Systematic Study of Transfer Learning Methodology

model architectures, where we found that encoder-decoder models generally outperformed "decoder-only" language models;

pre-training objectives, where we confirmed that fill-in-the-blank-style denoising objectives (where the model is trained to recover missing words in the input) worked best and that the most important factor was the computational cost;

unlabeled datasets, where we showed that training on in-domain data can be beneficial but that pre-training on smaller datasets can lead to detrimental overfitting;

training strategies, where we found that multitask learning could be close to competitive with a pre-train-then-fine-tune approach but requires carefully choosing how often the model is trained on each task;

and scale, where we compare scaling up the model size, the training time, and the number of ensembled models to determine how to make the best use of fixed compute power.

Insights + Scale = State-of-the-ArtGoogle Cloud TPU acceleratorsGLUESuperGLUESQuADCNN/Daily MailExtensionsClosed-Book Question AnsweringHurricane ConnieOn what date did Hurricane Connie occur?Colabpaperwithout access to any external knowledgeopen-domain question answering

During pre-training, T5 learns to fill in dropped-out spans of text (denoted by <M>) from documents in C4. To apply T5 to closed-book question answer, we fine-tuned it to answer questions without inputting any additional information or context. This forces T5 to answer questions based on “knowledge” that it internalized during pre-training.

TriviaQAWebQuestionsNatural Questions

Fill-in-the-Blank Text GenerationGPT-2Talk To TransformerAI Dungeonsized fill-in-the-blankI like to eat peanut butter and _4_ sandwiches,I love peanut butter and _N_ sandwiches,.fill{color:#4184F3;font-style:italic}.monospace{font-family:'Roboto Mono',monospace}.tabs{width:100%!important;margin:0!important;padding:0!important;list-style:none;position:relative;height:200px!important;margin-bottom:24px!important}.tabs li{display:inline-block;margin:0!important;padding:0!important;line-height:1!important}.tabs input[type="radio"]{position:absolute;top:0;left:0;opacity:0}.tabs label{display:block;padding:8px;width:50px;cursor:pointer;border-bottom:2px solid white;text-align:center;transition:.2s all ease-in-out}.tabs [id^="tab"]:checked+label{border-bottom:2px solid #4184F3;background:#e3edfd;color:#4184F3;font-weight:bold}.tabs label:hover{color:#4184F3;border-bottom:2px solid #4184F3;xbackground:#e6e9ff}.tabs .tab-content{z-index:2;display:none;overflow:auto;width:100%;height:160px;margin-top:0;padding:8px;position:absolute;left:0;line-height:1.5;border:1px solid #4184F3;border-top-color:#e6e9ff;border-left-color:#e6e9ff}.tabs .tab-content>p{margin-top:0}.tabs [id^="tab"]:checked ~ .tab-content{display:block;box-sizing:border-box;}@media only screen and (max-width:680px){.tabs .tab-content{margin-top:40px!important}.tabs{height:240px!important}}@media only screen and (max-width:500px){#content7,#content6{margin-top:0!important}}@media only screen and (max-width:460px){#content5{margin-top:0!important}}@media only screen and (max-width:400px){#content7,#content6,#content5{margin-top:0!important}}@media only screen and (max-width:360px){#content4{margin-top:0!important}}

N=1

I love peanut butter and jelly sandwiches.

N=2

I love peanut butter and jelly on my sandwiches.

N=4

I love peanut butter and jelly, which is what makes good sandwiches.

N=8

I love peanut butter and jelly, Yum! You can’t beat peanut butter and jelly sandwiches.

N=16

I love peanut butter and bread. Thanks!! This looks delicious. I love all types of peanut butter, but especially peanut butter/jam sandwiches.

N=32

I love peanut butter and banana sandwiches and the Peanut Butter Chocolate Chip Cookie Bites are now very easy to prepare. The best part is that the PB&J Cookie Bites are the perfect size for kid lunch boxes and everyone will enjoy them. Kids love these sandwiches.

N=64

I love peanut butter and jelly! It's like an incredible childhood love now that I'm in my 30s. I'm obsessed with peanut butter and jelly sandwiches. I have made this recipe many times with great success. I'm guessing you could probably find that it's from a 1950s cookbook. I haven't seen it listed in a current one but it is in a 1930 version (they seem to have retired it). Hope you enjoy it ;) I hope you'll post what you do with your sandwiches.

N=512

I love peanut butter and banana sandwiches. Peanut butter and bananas were the original sandwich spread (also known as PB&J or Peanut Butter and Jelly), so they are probably my favorite. When I was a kid, I knew what peanut butter and bananas tasted like, but I didn't really think of them as one flavor. I did recognize PB & J's as just a sandwich spread, and not really two separate flavours. When I was in graduate school researching food allergies and intolerances, PB&B is one of the main themes that came up. PB&B was and still is the favorite sandwich spread on my school cafeteria tray, and it's still on my personal board in the kitchen. I've seen PB&B on more than one occasion over the years. I can always find one of the other types of peanut butter spread at the school cafeteria and I find it's the perfect spread for PB&B. You have several options for making your own peanut butter (I love this recipe from Cookie and Kate) or buying it at stores like Trader Joes, Publix, and Safeway. The flavor of PB&B varies, but I've found that white chocolate is my favorite to use. This simple sandwich is perfect for lunch or an afternoon snack. When you're on the go, you can use a store bought product, but it just wouldn't taste as good. These peanut butter n' banana sandwich spreads (as well as other peanut butter spreads) are great for on the go eating, but they can also be refrigerated for a few hours and then served at your next party. Preheat oven to 350 degrees. Using a kitchen timer, or using a microwave, heat butter in a saucepan and melt over low heat. Add vanilla extract and milk and heat until nearly boiling. Add peanut butter, bananas, sugar, salt, and pepper, mixing well to combine all ingredients. Spread peanut butter mixture on both sides of your bread. Top with banana slices and a slice of cheese. Toast the bread until both sides are golden brown. Remove from toast and sprinkle a little more sugar and peanut butter on the top. Enjoy while the bread is warm. Store in an airtight container up to one day. Assemble peanut butter and banana sandwich spread by spreading the peanut butter mixture on each slice of bread. Add a banana slice on top and then a PB & J sandwich. Enjoy while the bread is still warm. P.S. You might also like these peanut butter and jelly sandwiches.

Conclusionfindingscodepre-trained modelsColab NotebookAcknowledgementsThis work has been a collaborative effort involving Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, Karishma Malkan, Noah Fiedel, and Monica Dinculescu.

Announcing the 2019 Google Faculty Research Award Recipients

Monday, February 24, 2020

Posted by Maggie Johnson, VP and Negar Saei, Program Manager, Google ResearchGoogle Faculty Research Awardsresearch philosophyMachine LearningSystemsHuman Computer Interactionwell-deserving recipientsour website

Setting Fairness Goals with the TensorFlow Constrained Optimization Library

Friday, February 21, 2020

Posted by Andrew Zaldivar, Responsible AI Advocate, Google Research, on behalf of the TFCO Teamloss functionfairnessTensorFlow Constrained Optimizationprecisionstrue positive ratesrecall ratesequalized oddspredictive parityHow Does TFCO Relate to Our AI Principles?AI PrinciplesWhat Are the Goals?Hardt et al.classifierpredictiondatasetlabelobjective functionalsofairness constraints

Illustration of a binary classification dataset with two protected groups: blue and orange. For ease of visualization, rather than plotting each individual data point, the densities are represented as ovals. The positive and negative signs denote the labels. The decision boundary drawn as a black dashed line separating positive predictions (regions above the line) and negative (regions below the line) labels, chosen to maximize accuracy.

unfair“equal opportunity” principle

Here the decision boundary is chosen to maximize the accuracy, subject to an equal opportunity (or true positive rate) constraint.

How Do I Know What Constraints To Set?contradictory constraintsfalse positive ratesin addition totrue positive ratesequalized odds

Here the decision boundary is chosen to maximize the accuracy, subject to both the true positive rate and false positive rate constraints.

eitherorCan It Fail?decreasingThe Curse of Small Sample Sizesstochastic gradientsWiki toxicity example

Comparison of unconstrained and constrained CNN models for classifying toxic comments on Wiki Talk pages.

Intersectionality – The Challenge of Fine Grained GroupsCelebA examplefalse positive rate

Comparison of unconstrained and constrained CNN models for classifying toxic comments on Wiki Talk pages.

Under the HoodcountsALT’19JMLR’19AAAI’20NeurIPS’19aICML’19TFCOtoxicity classificationsmile detectionAcknowledgementsThis work was a collaborative effort by the authors of TFCO and associated research papers, including Andrew Cotter, Maya R. Gupta, Heinrich Jiang, Harikrishna Narasimhan, Taman Narayan, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You.

Generating Diverse Synthetic Medical Image Data for Training Machine Learning Models

Wednesday, February 19, 2020

Posted by Timo Kohlberger and Yuan Liu, Software Engineers, Google HealthprogressGenerating Diverse Images of Skin ConditionsDermGAN: Synthetic Generation of Clinical Skin Images with PathologyMachine Learning for HealthNeurIPS 2019dermatology imagesFitzpatrick skin typesrelative few cases at the “boundaries”pix2pixgenerative adversarial networkU-Netcheckerboard artifacts.

Overview of the generator component of DermGAN. The model takes an RGB semantic map (red box) annotated with the skin condition's size and location (smaller orange rectangle), and outputs a realistic skin image. Colored boxes represent various neural network layers, such as convolutions and ReLU; the skip connections resemble the U-Net and enable information to be propagated at the appropriate scales.

The top row shows generated synthetic examples and the bottom row illustrates real images of basal cell carcinoma (left) and melanocytic nevus (right). More examples can be found in the paper.

DermGAN can be used to generate skin images (all with melanocytic nevus in this case) with different background skin types (top, by changing the input skin color) and different-sized lesions (bottom, by changing the input lesion size). As the input skin color changes, the lesion changes appearance to match what the lesion would look like on different skin types.

Generating Pathology Images with Different Labels Across Diverse Scannersmetastatic breast cancer detection algorithmsWhole-slide image focus quality: Automatic assessment and impact on AI cancer detectionJournal of Pathology Informatics

A sequence of images showing step-wise out-of-focus image generation. Images are shown in grayscale to accentuate the difference between steps. First, an in-focus image is collected (a) and a bokeh effect is added to produce a blurry image (b). Next, sensor noise is added to simulate a real image sensor (c), and finally JPEG compression is added to simulate the sharp edges introduced by post-acquisition software processing (d). A real out-of-focus image is shown for comparison (e).

modeling each step is essential

An example of a particularly interesting out-of-focus pattern across a biological tissue slice. Areas in blue were recognized by the model to be in-focus, whereas areas highlighted in yellow, orange, or red were more out of focus. The gradation in focus here (represented by concentric circles: a red/orange out-of-focus center surrounded by green/cyan mildly out-of-focus, and then a blue in-focus ring) was caused by a hard “stone” in the center that lifted the surrounding biological tissue.

Implications and Future Outlooktrainingvalidationonlyinverse probability weightingour work on ML for chest X-raysAcknowledgementsThese projects involved the efforts of multidisciplinary teams of software engineers, researchers, clinicians and cross functional contributors. Key contributors to these projects include Timo Kohlberger, Yun Liu, Melissa Moran, Po-Hsuan Cameron Chen, Trissia Brown, Jason Hipp, Craig Mermel, Martin Stumpe, Amirata Ghorbani, Vivek Natarajan, David Coz, and Yuan Liu. The authors would also like to acknowledge Daniel Fenner, Samuel Yang, Susan Huang, Kimberly Kanada, Greg Corrado and Erica Brand for their advice, members of the Google Health dermatology and pathology teams for their support, and Ashwin Kakarla and Shivamohan Reddy Garlapati for their team for image labeling.

AutoFlip: An Open Source Framework for Intelligent Video Reframing

Thursday, February 13, 2020

Posted by Nathan Frey, Senior Software Engineer, Google Research, Los Angeles and Zheng Sun, Senior Software Engineer, Google Research, Mountain Viewincreasing number of usersAutoFlip MediaPipe

Left: Original video (16:9). Middle: Reframed using a standard central crop (9:16). Right: Reframed with AutoFlip (9:16). By detecting the subjects of interest, AutoFlip is able to avoid cropping off important visual content.

AutoFlip Overview

Shot (Scene) DetectionVideo Content Analysisfaceobject detection modelsTensorFlow Lite

Left: People detection on sports footage. Right: Two face boxes (‘core’ and ‘all’ face landmarks). In narrow portrait crop cases, often only the core landmark box can fit.

ReframingstationarypanningtrackingstationarypantrackingEuclidean-normlow-degree polynomial

Top: Camera paths resulting from following the bounding boxes from frame-to-frame. Bottom: Final smoothed camera paths generated using Euclidean-norm path formation. Left: Scene in which objects are moving around, requiring a tracking camera path. Right: Scene where objects stay close to the same position; a stationary camera covers the content for the full duration of the scene.

best-effortrequiredletterbox effect

AutoFlip Use Cases

What’s Next?deep uncropAcknowledgmentsWe would like to thank our colleagues who contributed to Autoflip, Alexander Panagopoulos, Jenny Jin, Brian Mulford, Yuan Zhang, Alex Chen, Xue Yang, Mickey Wang, Justin Parra, Hartwig Adam, Jingbin Wang, and Weilong Yang; MediaPipe team who helped with open sourcing, Jiuqiang Tang, Tyler Mullen, Mogan Shieh, Ming Guang Yong, and Chuo-Ling Chang.

Learning to See Transparent Objects

Wednesday, February 12, 2020

Posted by Shreeyak Sajjan, Research Engineer, Synthesis AI and Andy Zeng, Research Scientist, Robotics at GoogleRGB-D camerasLIDARautonomous manipulatorsLambertian

Transparent objects often fail to be detected by optical 3D sensors. Top, Right: For instance, glass bottles do not show up in the 3D depth imagery captured from an Intel® RealSense™ D415 RGB-D camera. Bottom: A 3D visualization via point clouds constructed from the depth image.

Synthesis AIColumbia UniversityClearGrasplarge-scale synthetic datasetprevious methodspick and place robot

ClearGrasp uses deep learning to recover accurate 3D depth data of transparent surfaces.

A Visual Dataset of Transparent ObjectsImageNetWikipediaBERTMatterport3DScanNetlarge-scale dataset of transparent objectssurface normals

Some example data of transparent objects from the ClearGrasp synthetic dataset.

Left: The real-world image capturing setup, Middle: Custom user interface enables precisely replacing each transparent object with a spray-painted duplicate, Right: Example of captured data.

The Challenge

Specular reflections on transparent objects create distinct features that vary based on the object shape and provide strong visual cues for estimating surface normals.

RGB-D 3D cameraThe ClearGrasp Algorithm

Overview of our method. The point cloud was generated using the output depth and is colored with its surface normals.

Matterport3DScanNet

Surface Normal estimation on real images when trained on a) Matterport3D and ScanNet only (MP+SN), b) our synthetic dataset only, and c) MP+SN as well as our synthetic dataset. Note how the model trained on MP+SN fails to detect the transparent objects. The model trained on only synthetic data picks up the real plastic bottles remarkably well, but fails for other objects and surfaces. When trained on both, our model gets the best of both worlds.

Resultsquantitative experimentsalternativemethodsproject webpage

Qualitative results on real images. Top two rows: results on known objects. Bottom two rows: results on novel objects. The point clouds, colored with their surface normals, are generated from the corresponding depth images.

grasping algorithmUR5 robot arm

Manipulation of novel transparent objects using ClearGrasp. Note the challenging conditions: textureless background, complex object shapes and the directional light causing confusing shadows and caustics (the patterns of light that occur when light rays are reflected or refracted from a surface).

Limitations & Future WorkcausticsClearGraspdatasetproject website GitHub repositoryAcknowledgementsThis research was done by Shreeyak Sajjan (Synthesis.ai), Matthew Moore (Synthesis.ai), Mike Pan (Synthesis.ai), Ganesh Nagaraja (Synthesis.ai), Johnny Lee, Andy Zeng, and Shuran Song (Columbia University). We would like to thank Ryan Hickman for managerial support, Ivan Krasin and Stefan Welker for fruitful technical discussions, Cameron (@camfoxmusic) for sharing 3D models of his potion bottles and Sharat Sajjan for helping with web design.