Google AI Blog

Fairness Indicators: Scalable Infrastructure for Fair ML Systems

Wednesday, December 11, 2019

Posted by Catherina Xu and Tulsee Doshi, Product Managers, Google Researchhighlighting AI principlesTensorFlow WorldFairness Indicatorsmodel cardsinteractive case studyJigsaw’sUnintended Bias in Toxicity datasetnow available in betaWhat is ML Fairness?representations

The Fairness Indicators Suite of Toolsconfidence intervals What-If Tool

Using Fairness Indicators to visualize metrics for fairness evaluation.

Clicking on a slice in Fairness Indicators will load all the data points in that slice inside the What-If Tool widget. In this case, all data points with the “female” label are shown.

pip package: Includes Tensorflow Model Analysis (TFMA), Fairness Indicators, Tensorflow Data Validation (TFDV), What-If Tool, and example Colabs:

Fairness Indicators Example Colab — an introduction to Fairness Indicators usage
Fairness Indicators for TensorBoard — a TensorBoard plug-in usage example
Fairness Indicators with TFHub Embeddings — a Colab that investigates the effects of different embeddings on downstream fairness metrics
Fairness Indicators with Cloud Vision API's Face Detection Model — a Colab showing how Fairness Indicators can be used to generate evaluation results for model cards

GitHub repository: Source code

Guidance for usage: Fairness is highly contextual, and it’s important to carefully think through each use case and potential implications for users. This document provides guidance for selecting groups and metrics, and highlights evaluation best practices.

Case Study: Interactive case study on using Fairness Indicators, showing how Jigsaw's Conversation AI team detects bias in a classification model using the Toxic Comment Classification dataset.

How To Use Fairness Indicators in Models TodayTensorFlow Extended

Access Fairness Indicators as part of the Evaluator component in TFX

Access Fairness Indicators in TensorBoard when evaluating other real-time metrics

Download the Fairness Indicators pip package, and use Tensorflow Model Analysis as a standalone tool

Use Model Agnostic TFMA to compute Fairness Indicators based on the output of any model

Fairness Indicators Case Study case studyintroductory videoUnintended Bias in Toxicity datasetWhat’s next?verticallyhorizontallyactive learningmin-diffFairness Indicators GitHub repothis linktfx@tensorflow.orgAcknowledgementsThe core team behind this work includes Christina Greer, Manasi Joshi, Huanming Fang, Shivam Jindal, Karan Shukla, Osman Aka, Sanders Kleinfeld, Alicia Chang, Alex Hanna, and Dan Nanas. We would also like to thank James Wexler, Mahima Pushkarna, Meg Mitchell and Ben Hutchinson for their contributions to the project.

Lessons Learned from Developing ML for Healthcare

Tuesday, December 10, 2019

Posted by Yun Liu, Research Scientist and Po-Hsuan Cameron Chen, Research Engineer, Google HealthTIMI Risk Scorea variety of medical applicationscomplex medical recordsimproving the accuracy of genomic sequencingcardiovascular risk factorsrefractive errorgrading diabetic eye diseasediagnosing metastatic breast cancerleverage ML-based tools in an interactive fashionNature MaterialsJournal of the American Medical AssociationJAMAHow to Develop Machine Learning Models for HealthcareHow to develop machine learning models for healthcarepdfNature Materials

Two examples of the translational process of developing, validating, and implementing ML models for healthcare based on our work in detecting diabetic eye disease and metastatic breast cancer.

Empowering Doctors to Better Understand Machine Learning for HealthcareUsers’ Guide to the Medical Literature: How to Read Articles that use Machine LearningJAMAmemorize the training dataset

The intersection of two fields: ML and healthcare creates ambiguity in the term “validation dataset”. An ML validation set is typically used to refer to the dataset used for hyperparameter tuning, whereas a “clinical” validation set is typically used for final evaluation. To reduce confusion, we have opted to refer to the (ML) validation set as the “tuning” set.

Future outlookAcknowledgementsKey contributors to these projects include Yun Liu, Po-Hsuan Cameron Chen, Jonathan Krause, and Lily Peng. The authors would like to acknowledge Greg Corrado and Avinash Varadarajan for their advice, and the Google Health team for their support.

Google at NeurIPS 2019

Monday, December 9, 2019

Posted by Andrew Helton, Editor, Google Research CommunicationsConference on Neural Information Processing SystemsML-based Flood ForecastingAI for Social GoodGoogle Research FootballGoogle Dataset SearchTF-AgentsblueNeurIPS BoardSamy Bengio, Corinna CortesNeurIPS Advisory BoardJohn C. Platt, Fernando Pereira, Dale SchuurmansNeurIPS Organizing CommitteeHugo LarochelleKatherine HellerNicolas Le RouxPablo Samuel CastroAmir Globerson, Claudio Gentile, Cordelia Schmid, Corinna Cortes, Dale Schuurmans, Elad Hazan, Honglak Lee, Mehryar Mohri, Peter Bartlett, Satyen Kale, Sergey Levine, Surya GanguliAfshin Rostamizadeh, Alex Kulesza, Amin Karbasi, Andrew Dai, Been Kim, Boqing Gong, Brainslav Kveton, Ce Liu, Charles Sutton, Chelsea Finn, Cho-Jui Hsieh, D Sculley, Danny Tarlow, David Held, Denny Zhou, Yann Dauphin, Dustin Tran, Hartmut Neven, Hossein Mobahi, Ilya Tolstikhin, Jasper Snoek, Jean-Philippe Vert, Jeffrey Pennington, Kevin Swersky, Kun Zhang, Kunal Talwar, Lihong Li, Manzil Zaheer, Marc G Bellemare, Marco Cuturi, Maya Gupta, Meg Mitchell, Minmin Chen, Mohammad Norouzi, Moustapha Cisse, Olivier Bachem, Qiang Liu, Rong Ge, Sanjiv Kumar, Sanmi Koyejo, Sebastian Nowozin, Sergei Vassilvitskii, Shivani Agarwal, Slav Petrov, Srinadh Bhojanapalli, Stephen Bach, Timnit Gebru, Tomer Koren, Vitaly Feldman, William Cohen, Yann Dauphin, Nicolas Le RouxNeurIPS Workshops Program CommitteeYann Dauphin, Honglak Lee, Sebastian Nowozin, Fernanda ViegasNeurIPS Invited TalkSocial IntelligenceBlaise Aguera y ArcasAccepted PapersMemory Efficient Adaptive OptimizationRohan Anil, Vineet Gupta, Tomer Koren, Yoram SingerStand-Alone Self-Attention in Vision ModelsNiki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jon ShlensHigh Fidelity Video Prediction with Large Neural NetsRuben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, Honglak LeeUnsupervised Learning of Object Structure and Dynamics from VideosMatthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, Honglak LeeGPipe: Efficient Training of Giant Neural Networks using Pipeline ParallelismYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyouk Joong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, Zhifeng ChenQuadratic Video InterpolationXiangyu Xu, Li Si-Yao, Wenxiu Sun, Qian Yin, Ming-Hsuan YangOnline Stochastic Shortest Path with Bandit Feedback and Unknown Transition FunctionAviv Rosenberg, Yishay MansourIndividual Regret in Cooperative Nonstochastic Multi-Armed BanditsYogev Bar-On, Yishay MansourLearning to ScreenAlon Cohen, Avinatan Hassidim, Haim Kaplan, Yishay Mansour, Shay MoranDualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution CorrectionsOfir Nachum, Yinlam Chow, Bo Dai, Lihong LiA Kernel Loss for Solving the Bellman EquationYihao Feng, Lihong Li, Qiang LiuAccurate Uncertainty Estimation and Decomposition in Ensemble LearningJeremiah Liu, John Paisley, Marithani-Anna Kioumourtzoglou, Brent CoullSaccader: Improving Accuracy of Hard Attention Models for VisionGamaleldin F. Elsayed, Simon Kornblith, Quoc V. LeInvertible Convolutional FlowMahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, Daniel DuckworthHypothesis Set Stability and GeneralizationDylan J. Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, Karthik SridharanBandits with Feedback Graphs and Switching CostsRaman Arora, Teodor V. Marinov, Mehryar MohriRegularized Gradient BoostingCorinna Cortes, Mehryar Mohri, Dmitry StorcheusLogarithmic Regret for Online ControlNaman Agarwal, Elad Hazan, Karan SinghSampled Softmax with Random Fourier FeaturesAnkit Singh Rawat, Jiecao Chen, Felix Yu, Ananda Theertha Suresh, Sanjiv KumarMultilabel Reductions: What is My Loss Optimising?Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, Sanjiv KumarMetaInit: Initializing Learning by Learning to InitializeYann N. Dauphin, Sam SchoenholzGeneralization Bounds for Neural Networks via Approximate Description LengthAmit Daniely, Elad GranotVariance Reduction of Bipartite Experiments through Correlation ClusteringJean Pouget-Abadie, Kevin Aydin, Warren Schudy, Kay Brodersen, Vahab MirrokniLikelihood Ratios for Out-of-Distribution DetectionJie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji LakshminarayananCan You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset ShiftYaniv Ovadia, Emily Fertig, Jie Jessie Ren, D. Sculley, Josh Dillon, Sebastian Nowozin, Zack Nado, Balaji Lakshminarayanan, Jasper SnoekSurrogate Objectives for Batch Policy Optimization in One-step Decision MakingMinmin Chen, Ramki Gummadi, Chris Harris, Dale SchuurmansGlobally Optimal Learning for Structured Elliptical LossesYoav Wald, Nofar Noy, Gal Elidan, Ami WieselDPPNet: Approximating Determinantal Point Processes with Deep NetworksZelda Mariet, Yaniv Ovadia, Jasper SnoekGraph Normalizing FlowsJenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, Kevin SwerskyWhen Does Label Smoothing Help?Rafael Muller, Simon Kornblith, Geoff HintonOn the Role of Inductive Bias From Simulation and the Transfer to the Real World: a new Disentanglement DatasetMuhammad Waleed Gondal, Manuel Wüthrich, Đorđe Miladinović, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, Stefan BauerOn the Fairness of Disentangled RepresentationsFrancesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, Olivier BachemAre Disentangled Representations Helpful for Abstract Visual Reasoning?Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, Olivier BachemDon’t Blame the ELBO! A Linear VAE Perspective on Posterior CollapseJames Lucas, George Tucker, Roger Grosse, Mohammad NorouziStabilizing Off-Policy Q-Learning via Bootstrapping Error ReductionAviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey LevineOptimizing Generalized Rate Metrics with Game EquilibriumHarikrishna Narasimhan, Andrew Cotter, Maya GuptaOn Making Stochastic Classifiers DeterministicAndrew Cotter, Harikrishna Narasimhan, Maya GuptaDiscrete Flows: Invertible Generative Models of Discrete DataDustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, Ben PooleGraph Agreement Models for Semi-Supervised LearningOtilia Stretcu, Krishnamurthy Viswanathan, Dana Movshovitz-Attias, Emmanouil Platanios, Andrew Tomkins, Sujith RaviA Robust Non-Clairvoyant Dynamic Mechanism for Contextual AuctionsYuan Deng, Sébastien Lahaie, Vahab MirrokniAdversarial Robustness through Local LinearizationChongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy (Dj) Dvijotham, Alhusein Fawzi, Soham De, Robert Stanforth, Pushmeet KohliA Geometric Perspective on Optimal Representations for Reinforcement LearningMarc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, Clare LyleOnline Learning via the Differential Privacy LensJacob Abernethy, Young Hun Jung, Chansoo Lee, Audra McMillan, Ambuj TewariReducing the Variance in Online Optimization by Transporting Past GradientsSébastien M. R. Arnold, Pierre-Antoine Manzagol, Reza Babanezhad, Ioannis Mitliagkas, Nicolas Le RouxUniversality and Individuality in Neural Dynamics Across Large Populations of Recurrent NetworksNiru Maheswaranathan, Alex Williams, Matt Golub, Surya Ganguli, David SussilloReverse Engineering Recurrent Networks for Sentiment Classification Reveals Line Attractor DynamicsNiru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, David SussilloStrategizing Against No-Regret LearnersYuan Deng, Jon Schneider, Balasubramanian SivanPrior-Free Dynamic Auctions with Low Regret BuyersYuan Deng, Jon Schneider, Balasubramanian SivanPrivate Stochastic Convex Optimization with Optimal RatesRaef Bassily, Vitaly Feldman, Kunal Talwar, Abhradeep ThakurtaComputational Separations between Sampling and OptimizationKunal TalwarMomentum-Based Variance Reduction in Non-Convex SGDAshok Cutkosky and Francesco OrabonaKernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise AccelerationKwang-Sung Jun, Ashok Cutkosky, Francesco OrabonaFast and Flexible Multi-Task Classification using Conditional Neural Adaptive ProcessesJames Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, Richard E. TurnerIcebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian ModelWenbo Gong, Sebastian Tschiatschek, Richard E. Turner, Sebastian Nowozin, Jose Miguel Hernandez-Lobato, Cheng ZhangMultiview Aggregation for Learning Category-Specific Shape ReconstructionSrinath Sridhar, Davis Rempe, Julien Valentin, Sofien Bouaziz, Leonidas J. GuibasVisualizing and Measuring the Geometry of BERTAndy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, Martin WattenbergLocality-Sensitive Hashing for f-Divergences: Mutual Information Loss and BeyondLin Chen, Hossein Esfandiari, Thomas Fu, Vahab S. MirrokniA Benchmark for Interpretability Methods in Deep Neural NetworksSara Hooker, Dumitru Erhan, Pieter-jan Kindermans, Been KimPractical and Consistent Estimation of f-DivergencesPaul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, Ilya TolstikhinTree-Sliced Variants of Wasserstein DistancesTam Le, Makoto Yamada, Kenji Fukumizu, Marco CuturiGame Design for Eliciting Distinguishable BehaviorFan Yang, Liu Leqi, Yifan Wu, Zachary Lipton, Pradeep Ravikumar, Tom M Mitchell, William CohenDifferentially Private Anonymized HistogramsAnanda Theertha SureshLocally Private Gaussian EstimationMatthew Joseph, Janardhan Kulkarni, Jieming Mao, Zhiwei Steven WuExponential Family Estimation via Adversarial Dynamics EmbeddingBo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, Dale SchuurmansLearning to Predict Without Looking Ahead: World Models Without Forward PredictionC. Daniel Freeman, Luke Metz, David HaAdaptive Density Estimation for Generative ModelsThomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, Jakob VerbeekWeight Agnostic Neural NetworksAdam Gaier, David HaRetrosynthesis Prediction with Conditional Graph Logic NetworkHanjun Dai, Chengtao Li, Connor Coley, Bo Dai, Le SongLarge Scale Structure of Neural Network Loss LandscapesStanislav Fort, Stainslaw JastrzebskiOff-Policy Evaluation via Off-Policy ClassificationAlex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, Sergey LevineDomes to Drones: Self-Supervised Active Triangulation for 3D Human Pose ReconstructionAleksis Pirinen, Erik Gartner, Cristian SminchisescuEnergy-Inspired Models: Learning with Sampler-Induced DistributionsDieterich Lawson, George Tucker, Bo Dai, Rajesh RanganathFrom Deep Learning to Mechanistic Understanding in Neuroscience: The Structure of Retinal PredictionHidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, Surya Ganguli
Language as an Abstraction for Hierarchical Deep Reinforcement LearningYiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea FinnBayesian Layers: A Module for Neural Network UncertaintyDustin Tran, Michael W. Dusenberry, Mark van der Wilk, Danijar HafnerAdaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty EstimatesHugo Penedones, Carlos Riquelme, Damien Vincent, Hartmut Maennel, Timothy Mann, Andre Barreto, Sylvain Gelly, Gergely NeuA Unified Framework for Data Poisoning Attack to Graph-based Semi-Supervised LearningXuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, Cho-Jui HsiehMixMatch: A Holistic Approach to Semi-Supervised LearningDavid Berthelot, Nicholas Carlini, Ian Goodfellow (work done while at Google), Avital Oliver, Nicolas Papernot, Colin RaffelSMILe: Scalable Meta Inverse Reinforcement Learning through Context-Conditional PoliciesSeyed Kamyar Seyed Ghasemipour, Shixiang (Shane) Gu, Richard ZemelLimits of Private Learning with Access to Public DataNoga Alon, Raef Bassily, Shay MoranRegularized Weighted Low Rank ApproximationFrank Ban, David Woodruff, Richard ZhangUnsupervised Curricula for Visual Meta-Reinforcement LearningAllan Jabri, Kyle Hsu, Abhishek Gupta, Benjamin Eysenbach, Sergey Levine, Chelsea FinnSecretary Ranking with Minimal InversionsSepehr Assadi, Eric Balkanski, Renato Paes LemeMixtape: Breaking the Softmax Bottleneck EfficientlyZhilin Yang, Thang Luong, Russ Salakhutdinov, Quoc V. LeBudgeted Reinforcement Learning in Continuous State SpaceNicolas Carrara, Edouard Leurent, Romain Laroche, Tanguy Urvoy, Odalric-Ambrym Maillard, Olivier PietquinFrom Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox OptimizationKrzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao TangFlattening a Hierarchical Clustering through Active LearningFabio Vitale, Anand Rajagopalan, Claudio GentileRobust Attribution RegularizationJiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, Somesh JhaRobustness Verification of Tree-based ModelsHongge Chen, Huan Zhang, Si Si, Yang Li, Duane Boning, Cho-Jui HsiehMeta Architecture SearchAlbert Shaw, Wei Wei, Weiyang Liu, Le Song, Bo DaiContextual Bandits with Cross-LearningSantiago Balseiro, Negin Golrezaei, Mohammad Mahdian, Vahab Mirrokni, Jon SchneiderDynamic Incentive-Aware Learning: Robust Pricing in Contextual AuctionsNegin Golrezaei, Adel Javanmard, Vahab MirrokniOptimizing Generalized Rate Metrics with Three PlayersHarikrishna Narasimhan, Andrew Cotter, Maya GuptaNoise-Tolerant Fair ClassificationAlexandre Louis Lamy, Ziyuan Zhong, Aditya Krishna Menon, Nakul VermaTowards Automatic Concept-based ExplanationsAmirata Ghorbani, James Wexler, James Zou, Been KimLocally Private Learning without Interaction Requires SeparationAmit Daniely, Vitaly FeldmanLearning GANs and Ensembles Using DiscrepancyBen Adlam, Corinna Cortes, Mehryar Mohri, Ningshan ZhangCondConv: Conditionally Parameterized Convolutions for Efficient InferenceBrandon Yang, Gabriel Bender, Quoc V. Le, Jiquan NgiamA Fourier Perspective on Model Robustness in Computer VisionDong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin GilmerRobust Bi-Tempered Logistic Loss Based on Bregman DivergencesEhsan Amid, Manfred K. Warmuth, Rohan Anil, Tomer KorenWhich Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic ModelGuodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, Roger GrosseWide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentJaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey PenningtonAbstract Reasoning with Distracting FeaturesKecheng Zheng, Zheng-Jun Zha, Wei WeiSearch on the Replay Buffer: Bridging Planning and Reinforcement LearningBenjamin Eysenbach, Ruslan Salakhutdinov, Sergey LevineDifferentiable Ranking and Sorting Using Optimal TransportMarco Cuturi, Olivier Teboul, Jean-Philippe VertXLNet: Generalized Autoregressive Pretraining for Language UnderstandingZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. LePrivate Learning Implies Online Learning: An Efficient ReductionAlon Gonen, Elad Hazan, Shay MoranEvaluating Protein Transfer Learning with TAPERoshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, Yun SongTight Dimensionality Reduction for Sketching Low Degree Polynomial KernelsMichela Meister, Tamas Sarlos, David P. WoodruffNo Pressure! Addressing the Problem of Local Minima in Manifold Learning AlgorithmsMax VladymyrovSubspace Detours: Building Transport Plans that are Optimal on Subspace ProjectionsBoris Muzellec, Marco CuturiOn the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement DatasetMuhammad Waleed Gondal, Manuel Wüthrich, Ðorde Miladinovíc, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, Stefan BauerStacked Capsule AutoencodersAdam R. Kosiorek, Sara Sabour, Yee Whye Teh, Geoffrey E. HintonWasserstein Dependency Measure for Representation LearningSherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, Pierre SermanetSampling Sketches for Concave Sublinear Functions of FrequenciesEdith Cohen, Ofir GeriHamiltonian Neural NetworksSam Greydanus, Misko Dzamba, Jason YosinskiComputational Mirrors: Blind Inverse Light Transport by Deep Matrix FactorizationMiika Aittala, Prafull Sharma, Lukas Murmann, Adam B. Yedidia, Gregory W. Wornell, William T. Freeman, Frédo DurandTransfusion: Understanding Transfer Learning for Medical ImagingMaithra Raghu, Chiyuan Zhang, Jon Kleinberg, Samy BengioDifferentially Private Covariance EstimationKareem Amin, Travis Dick, Alex Kulesza, Andres Munoz, Sergei VassilvitskiiLearning Transferable Graph ExplorationHanjun Dai, Yujia Li, Chenglong Wang, Rishabh Singh, Po-Sen Huang, Pushmeet KohliNeural Attribution for Semantic Bug-Localization in Student ProgramsRahul Gupta, Aditya Kanade, Shirish ShevadePyTorch: An Imperative Style, High-Performance Deep Learning LibraryAdam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith ChintalaBreaking the Glass Ceiling for Embedding-Based Classifiers for Large Output SpacesChuan Guo, Ali Mousavi, Xiang Wu, Daniel Holtmann-Rice, Satyen Kale, Sashank Reddi, Sanjiv KumarEfficient Rematerialization for Deep NetworksRavi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, Joshua R. WangWorkshops3rd Conversational AI: Today's Practice and Tomorrow's PotentialBill ByrneAI for Humanitarian Assistance and Disaster Response WorkshopYossi MatiasBayesian Deep LearningKevin P MurphyBeyond First Order Methods in Machine Learning SystemsElad HazanBiological and Artificial Reinforcement LearningIgor MordatchContext and Compositionality in Biological and Artificial Neural SystemsKenton LeeDeep Reinforcement LearningChelsea FinnDocument IntelligenceTania Bedrax WeissFederated Learning for Data Privacy and ConfidentialityJakub KonečnýBrendan McMahanFrançoise Beaufays, Daniel RamageGraph Representation LearningRianne van den BergHuman-Centric Machine LearningBeen KimInformation Theory and Machine LearningBen PooleAlex AlemiKR2ML - Knowledge Representation and Reasoning Meets Machine LearningWilliam CohenLearning Meaningful Representations of LifeJasper SnoekAlexander WiltschkoLearning Transferable SkillsDavid HaMachine Learning for Creativity and DesignAdam RobertsJesse EngelMachine Learning for Health (ML4H): What Makes Machine Learning in Medicine Different?Lily Peng, Alan Karthikesalingam, Dale WebsterMachine Learning and the Physical SciencesYasaman Bahri, Samual SchoenholzML for SystemsMilad HashemiKevin SwerskyAzalia MirhoseiniAnna GoldieJeff DeanOptimal Transport for Machine LearningMarco CuturiThe Optimization Foundations of Reinforcement LearningBo DaiNicolas Le RouxLihong LiDale SchuurmansPrivacy in Machine LearningBrendan McMahanProgram Transformations for MLPascal LamblinAlexander WiltschkoBart van MerrienboerEmily FertigSkye Wanderman-MilneReal Neurons & Hidden Units: Future Directions at the Intersection of Neuroscience and Artificial IntelligenceDavid SussilloRobot Learning: Control and Interaction in the Real WorldStefan SchaalSafety and Robustness in Decision MakingYinlam ChowScience Meets Engineering of Deep LearningYasaman Bahri, Surya Ganguli‎, Been Kim, Surya GanguliSets and PartitionsManzil ZaheerAndrew McCallumAmr AhmedTackling Climate Change with MLJohn PlattJeff DeanVisually Grounded Interaction and LanguageJason BaldridgeWorkshop on Machine Learning with GuaranteesMehryar MohriTutorialsRepresentation Learning and FairnessMoustapha Cisse, Sanmi Koyejo

Understanding Transfer Learning for Medical Imaging

Friday, December 6, 2019

Posted by Maithra Raghu and Chiyuan Zhang, Research Scientists, Google Researchtransfer learningpretrainingImageNetfine-tuningmedical imaginginterpreting chest x-raysidentifying eye diseasesearly detection of Alzheimer’s diseasethe effects on performance improvementcontribution of the underlying architecturepretraining dataset type and sizeNeurIPS 2019Transfusion: Understanding Transfer Learning for Medical Imagingfeature independentPerformance Evaluationdiabetic retinopathyfundus photographsfive different diseases from chest x-raysResNet50Inception-v3convolutional neural networksconvolution-batchnorm-ReLU

Surprisingly, transfer learning does not significantly affect performance on medical imaging tasks, with models trained from scratch performing nearly as well as standard ImageNet transferred models.

On the medical imaging tasks, the much smaller CBR models perform at a level comparable to the standard ImageNet architectures.

As the CBR models are much smaller and shallower than the standard ImageNet models, they perform much worse on ImageNet classification, highlighting that ImageNet performance is not indicative of performance on medical tasks.

The two medical tasks are much smaller in size than ImageNet (~200k vs ~1.2m training images), but in the very small data regime, there may only be a few thousand training examples. We evaluated transfer learning in this very small data regime, finding that while there was a larger gap in performance between transfer and training from scratch for large models (ResNet) this was not true for smaller models (CBRs), suggesting that the large models designed for ImageNet might be too overparameterized for the very small data regime.

Representation Analysishidden representationsrecentmethodsingular vector canonical correlation analysiscode and tutorialscanonical correlation analysissimilarity score

Representation similarity scores between networks trained from random initialization and networks trained from pretrained ImageNet weights (orange), and baseline similarity scores of representations trained from two different random initializations (blue). Higher values indicate greater similarity. For larger models, representations learned from random initialization are much more similar to each other than those learned through transfer. This is not the case for smaller models.

even

Effects on Convergence: Feature Independent Benefits and Hybrid Approachesfeature independentweight scalingMean Var Init

Filter visualization of weights initialized according to pretrained ImageNet weights, Random Init, and Mean Var Init. Only the ImageNet Init filters have pretrained (Gabor-like) structure, as Rand Init and Mean Var weights are iid.

weight transfusion

Learning curves comparing the convergence speed with AUC on the test set. Using only the scaling of the pretrained weights (Mean Var Init) helps with convergence speed. The figures compare the standard transfer learning and the Mean Var initialization scheme to training from random initialization.

hybrid approaches

Hybrid approaches to transfer learning on Resnet50 (left) and CBR models (right) — reusing a subset of the weights and slimming the remainder of the network (Slim), and using mathematically synthesized Gabors for conv1 (Synthetic Gabor).

mathematically synthesized Gabor filters

Synthetic Gabor filters used to initialize the first layer if neural networks in some of the experiments in this paper. The Gabor filters are generated as grayscale images and repeated across the RGB channels. Left: Low frequencies. Right: High frequencies.

Conclusion and Open QuestionsAcknowledgementsSpecial thanks to Samy Bengio and Jon Kleinberg, who are co-authors on this work. Thanks also to Geoffrey Hinton for helpful feedback.

Developing Deep Learning Models for Chest X-rays with Adjudicated Image Labels

Tuesday, December 3, 2019

Posted by Dave Steiner, MD, Research Scientist and Shravya Shetty, Technical Lead, Google Healthearly-stage lung cancerspneumothoraceslung cancer detectionprostate cancer gradingdifferential diagnosesvaluableeffortsChest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted EvaluationRadiologyhereData OverviewApollo HospitalsChestX-ray14National Institutes of Healthhas limitations

Chest X-ray depicting an upper left lobe pneumothorax identified by the model and the adjudication panel, but missed by the individual radiologist readers. Left: The original image. Right: The same image with the most important regions for the model prediction highlighted in orange.

Training Set Labels Using Deep Learning and Visual Image ReviewCreating and Sharing Improved Reference Standard Labelssharing with the research communityFuture OutlookAcknowledgementsKey contributors to this project at Google include Sid Mittal, Gavin Duggan, Anna Majkowska, Scott McKinney, Andrew Sellergren, David Steiner, Krish Eswaran, Po-Hsuan Cameron Chen, Yun Liu, Shravya Shetty, and Daniel Tse. Significant contributions and input were also made by radiologist collaborators Joshua Reicher, Alexander Ding, and Sreenivasa Raju Kalidindi. The authors would also like to acknowledge many members of the Google Health radiology team including Jonny Wong, Diego Ardila, Zvika Ben-Haim, Rory Sayres, Shahar Jamshy, Shabir Adeel, Mikhail Fomitchev, Akinori Mitani, Quang Duong, William Chen and Sahar Kazemzadeh. Sincere appreciation also goes to the many radiologists who enabled this work through their expert image interpretation efforts throughout the project.

Blog

Fairness Indicators: Scalable Infrastructure for Fair ML Systems

Lessons Learned from Developing ML for Healthcare

Google at NeurIPS 2019

Understanding Transfer Learning for Medical Imaging

Developing Deep Learning Models for Chest X-rays with Adjudicated Image Labels

Labels

Archive

Feed