Google AI Blog: December 2019

ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Friday, December 20, 2019

Posted by Radu Soricut and Zhenzhong Lan, Research Scientists, Google ResearchBERTnatural language processingALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsICLR 2020Stanford Question Answering DatasetSATRACEreleasedTensorFlowWhat Contributes to NLP Performance?independentdependentSQuAD2.0RACETransformerBERTXLNetRoBERTaattention-feedforwardSQuAD2.0RACESQuAD2.0RACEOptimized Model Performance with the RACE DatasetSAT Reading TestRACE datasetXLNetRoBERTa

Machine performance on the RACE challenge (SAT-like reading comprehension). A random-guess baseline score is 25.0. The maximum possible score is 95.0.

open-sourcing ALBERT

The On-Device Machine Learning Behind Recorder

Wednesday, December 18, 2019

Posted by Itay Inbar and Nir Shemy, Software Engineers, Google ResearchRecorder Transcriptionimprovements announced earlier this year

Recording Content Visualization via Sound Classification using CNNs to classify audio soundsdatasets for audio event detection

Suggesting Tags for Titles

unigram and bigram

Tags extraction pipeline architecture

ConclusionAcknowledgmentsSpecial thanks to Dror Ayalon who played a key role in developing and forming the above features and without whom this blog post wouldn’t have been possible. We would also want to thank all our team members and collaborators who worked on this project with us: Amit Pitaru, Kelsie Van Deman, Isaac Blankensmith, Teo Soares, John Watkinson, Matt Hall, Josh Deitel, Benny Schlesinger, Yoni Tsafir, Michelle Tadmor Ramanovich, Danielle Cohen, Sushant Prakash, Renat Aksitov, Ed West, Max Gubin, Tiantian Zhang, Aaron Cohen, Yunhsuan Sung, Chung-Ching Chang, Nathan Dass, Amin Ahmad, Tiago Camolesi, Guilherme Santos‎, Julio da Silva, Dan Ellis, Qiao Liang, Arun Narayanan‎, Rohit Prabhavalkar, Benyah Shaparenko‎, Alex Salcianu, Mike Tsao, Shenaz Zack, Sherry Lin, James Lemieux, Jason Cho, Thomas Hall‎, Brian Chen, Allen Su, Vincent Peng‎, Richard Chou‎, Henry Liu‎, Edward Chen, Yitong Lin, Tracy Wu, Yvonne Yang‎.

Improving Out-of-Distribution Detection in Machine Learning Models

Tuesday, December 17, 2019

Posted by Jie Ren, Research Scientist, Google Research and Balaji Lakshminarayanan, Research Scientist, DeepMindout-of-distribution

New bacterial classes are gradually discovered over the years. A classifier trained on known classes achieves high accuracy for test inputs belonging to known classes, but can wrongly classify inputs from unknown classes (i.e., out-of-distribution) into known classes with high confidence.

Likelihood Ratios for Out-of-Distribution DetectionNeurIPS 2019releasedlikelihoodlikelihood-ratioWhy Do Density Models Fail At OOD Detection?NCBI catalog of prokaryotic genome sequencesearlierworkPixelCNN++Fashion-MNISTMNIST

Left: Histogram of likelihood values for in- and out-of-distribution (OOD) genomic sequences. The likelihood fails to separate in-distribution and OOD genomic sequences. Right: A similar plot for a model trained on Fashion-MNIST and evaluated on MNIST. The model assigns higher likelihood values for OOD (MNIST) than in-distribution images.

backgroundsemantic

Left top: Sample images from Fashion-MNIST. Left bottom: Sample images from MNIST. Right: Background and semantic components in an MNIST image.

Likelihood Ratios For OOD Detectionlikelihood ratio method nucleotidesbackground contrastive score

Left: Log-likelihood heatmaps for Fashion-MNIST and MNIST datasets. Right: The same examples showing heatmaps of the likelihood-ratio. Pixels with higher values are of lighter shades. The likelihood is dominated by the “background” pixels, whereas the likelihood ratio focuses on the “semantic” pixels and is thus better for OOD detection.

AUROC scorepaperGitHub repositoryAcknowledgmentsThe work described here was authored by Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji Lakshminarayanan, through a collaboration spanning several teams across Google AI and DeepMind. We are grateful for all the discussions and feedback on this work that we received from the reviewers at NeurIPS 2019, and our colleagues at Google and DeepMind: Alexander A. Alemi, Andreea Gane, Brian Lee, D. Sculley, Eric Jang, Jacob Burnim, Katherine Lee, Matthew D. Hoffman, Noah Fiedel, Rif A. Saurous, Suman Ravuri, Thomas Colthurst, Yaniv Ovadia, along with the Google Brain and TensorFlow teams.

Improvements to Portrait Mode on the Google Pixel 4 and Pixel 4 XL

Monday, December 16, 2019

Posted by Neal Wadhwa, Software Engineer and Yinda Zhang, Research Scientist, Google ResearchPortrait Modemachine learning to estimate depthdual-pixel auto-focusbokehSLR

Pixel 4’s Portrait Mode allows for Portrait Shots at both near and far distances and has SLR-like background blur. (Photos Credit: Alain Saal-Dalma and Mike Milne)

A Short Recap

The dual-pixel views of the bulb have much more parallax than the views of the man because the bulb is much closer to the camera.

parallaxcorrespondingDual Cameras are Complementary to Dual-Pixels

Left: Dual-pixel views. Right: Dual-camera views. The dual-pixel views have only a subtle vertical parallax in the background, while the dual-camera views have much greater horizontal parallax. While this makes it easier to estimate depth in the background, some pixels to the man’s right are visible in only the primary camera’s view making it difficult to estimate depth there.

aperture problemblog postDepth from Dual Cameras and Dual-Pixelslast yearTensorflowconvolutional neural networkencoders

Our network to predict depth from dual-pixels and dual-cameras. The network uses two encoders, one for each input and a shared decoder with skip connections and residual blocks.

drop-out

Depth maps from our network where either only one input is provided or both are provided. Top: The two inputs provide depth information for lines in different directions. Bottom: Dual-pixels provide better depth in the regions visible in only one camera, emphasized in the insets. Dual-cameras provide better depth in the background and ground. (Photo Credit: Mike Milne)

hereSLR-Like Bokehbokeh

Left: SLRs produce high contrast bokeh disks. Middle: It is hard to make out the disks in our old background blur. Right: Our new bokeh is closer to that of an SLR.

HDR+album

Blurring before tone mapping improves the look of the backgrounds by making it more saturated and by making disks higher contrast.

Try it YourselfAcknowledgmentsThis work wouldn’t have been possible without Rahul Garg, Sergio Orts Escolano, Sean Fanello, Christian Haene, Shahram Izadi, David Jacobs, Alexander Schiffhauer Yael Pritch Knaan and Marc Levoy. We would also like to thank the Google Camera team for helping to integrate these algorithms into the Pixel 4. Special thanks to our photographers Mike Milne, Andy Radin, Alain Saal-Dalma, and Alvin Li who took numerous test photographs for us.

Fairness Indicators: Scalable Infrastructure for Fair ML Systems

Wednesday, December 11, 2019

Posted by Catherina Xu and Tulsee Doshi, Product Managers, Google Researchhighlighting AI principlesTensorFlow WorldFairness Indicatorsmodel cardsinteractive case studyJigsaw’sUnintended Bias in Toxicity datasetnow available in betaWhat is ML Fairness?representations

The Fairness Indicators Suite of Toolsconfidence intervals What-If Tool

Using Fairness Indicators to visualize metrics for fairness evaluation.

Clicking on a slice in Fairness Indicators will load all the data points in that slice inside the What-If Tool widget. In this case, comments with the “buddhist” label are shown, including those with additional religion labels.

pip package: Includes Tensorflow Model Analysis (TFMA), Fairness Indicators, Tensorflow Data Validation (TFDV), What-If Tool, and example Colabs:

Fairness Indicators Example Colab — an introduction to Fairness Indicators usage
Fairness Indicators for TensorBoard — a TensorBoard plug-in usage example
Fairness Indicators with TFHub Embeddings — a Colab that investigates the effects of different embeddings on downstream fairness metrics
Fairness Indicators with Cloud Vision API's Face Detection Model — a Colab showing how Fairness Indicators can be used to generate evaluation results for model cards

GitHub repository: Source code

Guidance for usage: Fairness is highly contextual, and it’s important to carefully think through each use case and potential implications for users. This document provides guidance for selecting groups and metrics, and highlights evaluation best practices.

Case Study: Interactive case study on using Fairness Indicators, showing how Jigsaw's Conversation AI team detects bias in a classification model using the Toxic Comment Classification dataset.

How To Use Fairness Indicators in Models TodayTensorFlow Extended

Access Fairness Indicators as part of the Evaluator component in TFX

Access Fairness Indicators in TensorBoard when evaluating other real-time metrics

Download the Fairness Indicators pip package, and use Tensorflow Model Analysis as a standalone tool

Use Model Agnostic TFMA to compute Fairness Indicators based on the output of any model

Fairness Indicators Case Study case studyintroductory videoUnintended Bias in Toxicity datasetWhat’s next?verticallyhorizontallyactive learningmin-diffFairness Indicators GitHub repothis linktfx@tensorflow.orgAcknowledgementsThe core team behind this work includes Christina Greer, Manasi Joshi, Huanming Fang, Shivam Jindal, Karan Shukla, Osman Aka, Sanders Kleinfeld, Alicia Chang, Alex Hanna, and Dan Nanas. We would also like to thank James Wexler, Mahima Pushkarna, Meg Mitchell, Andrew Zaldivar and Ben Hutchinson for their contributions to the project.

Lessons Learned from Developing ML for Healthcare

Tuesday, December 10, 2019

Posted by Yun Liu, Research Scientist and Po-Hsuan Cameron Chen, Research Engineer, Google HealthTIMI Risk Scorea variety of medical applicationscomplex medical recordsimproving the accuracy of genomic sequencingcardiovascular risk factorsrefractive errorgrading diabetic eye diseasediagnosing metastatic breast cancerleverage ML-based tools in an interactive fashionNature MaterialsJournal of the American Medical AssociationJAMAHow to Develop Machine Learning Models for HealthcareHow to develop machine learning models for healthcarepdfNature Materials

Two examples of the translational process of developing, validating, and implementing ML models for healthcare based on our work in detecting diabetic eye disease and metastatic breast cancer.

Empowering Doctors to Better Understand Machine Learning for HealthcareUsers’ Guide to the Medical Literature: How to Read Articles that use Machine LearningJAMAmemorize the training dataset

The intersection of two fields: ML and healthcare creates ambiguity in the term “validation dataset”. An ML validation set is typically used to refer to the dataset used for hyperparameter tuning, whereas a “clinical” validation set is typically used for final evaluation. To reduce confusion, we have opted to refer to the (ML) validation set as the “tuning” set.

Future outlookAcknowledgementsKey contributors to these projects include Yun Liu, Po-Hsuan Cameron Chen, Jonathan Krause, and Lily Peng. The authors would like to acknowledge Greg Corrado and Avinash Varadarajan for their advice, and the Google Health team for their support.

Google at NeurIPS 2019

Monday, December 9, 2019

Posted by Andrew Helton, Editor, Google Research CommunicationsConference on Neural Information Processing SystemsML-based Flood ForecastingAI for Social GoodGoogle Research FootballGoogle Dataset SearchTF-AgentsblueNeurIPS BoardSamy Bengio, Corinna CortesNeurIPS Advisory BoardJohn C. Platt, Fernando Pereira, Dale SchuurmansNeurIPS Organizing CommitteeHugo LarochelleKatherine HellerNicolas Le RouxPablo Samuel CastroAmir Globerson, Claudio Gentile, Cordelia Schmid, Corinna Cortes, Dale Schuurmans, Elad Hazan, Honglak Lee, Mehryar Mohri, Peter Bartlett, Satyen Kale, Sergey Levine, Surya GanguliAfshin Rostamizadeh, Alex Kulesza, Amin Karbasi, Andrew Dai, Been Kim, Boqing Gong, Brainslav Kveton, Ce Liu, Charles Sutton, Chelsea Finn, Cho-Jui Hsieh, D Sculley, Danny Tarlow, David Held, Denny Zhou, Yann Dauphin, Dustin Tran, Hartmut Neven, Hossein Mobahi, Ilya Tolstikhin, Jasper Snoek, Jean-Philippe Vert, Jeffrey Pennington, Kevin Swersky, Kun Zhang, Kunal Talwar, Lihong Li, Manzil Zaheer, Marc G Bellemare, Marco Cuturi, Maya Gupta, Meg Mitchell, Minmin Chen, Mohammad Norouzi, Moustapha Cisse, Olivier Bachem, Qiang Liu, Rong Ge, Sanjiv Kumar, Sanmi Koyejo, Sebastian Nowozin, Sergei Vassilvitskii, Shivani Agarwal, Slav Petrov, Srinadh Bhojanapalli, Stephen Bach, Timnit Gebru, Tomer Koren, Vitaly Feldman, William Cohen, Yann Dauphin, Nicolas Le RouxNeurIPS Workshops Program CommitteeYann Dauphin, Honglak Lee, Sebastian Nowozin, Fernanda ViegasNeurIPS Invited TalkSocial IntelligenceBlaise Aguera y ArcasAccepted PapersMemory Efficient Adaptive OptimizationRohan Anil, Vineet Gupta, Tomer Koren, Yoram SingerStand-Alone Self-Attention in Vision ModelsNiki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jon ShlensHigh Fidelity Video Prediction with Large Neural NetsRuben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, Honglak LeeUnsupervised Learning of Object Structure and Dynamics from VideosMatthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, Honglak LeeGPipe: Efficient Training of Giant Neural Networks using Pipeline ParallelismYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyouk Joong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, Zhifeng ChenQuadratic Video InterpolationXiangyu Xu, Li Si-Yao, Wenxiu Sun, Qian Yin, Ming-Hsuan YangOnline Stochastic Shortest Path with Bandit Feedback and Unknown Transition FunctionAviv Rosenberg, Yishay MansourIndividual Regret in Cooperative Nonstochastic Multi-Armed BanditsYogev Bar-On, Yishay MansourLearning to ScreenAlon Cohen, Avinatan Hassidim, Haim Kaplan, Yishay Mansour, Shay MoranDualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution CorrectionsOfir Nachum, Yinlam Chow, Bo Dai, Lihong LiA Kernel Loss for Solving the Bellman EquationYihao Feng, Lihong Li, Qiang LiuAccurate Uncertainty Estimation and Decomposition in Ensemble LearningJeremiah Liu, John Paisley, Marithani-Anna Kioumourtzoglou, Brent CoullSaccader: Improving Accuracy of Hard Attention Models for VisionGamaleldin F. Elsayed, Simon Kornblith, Quoc V. LeInvertible Convolutional FlowMahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, Daniel DuckworthHypothesis Set Stability and GeneralizationDylan J. Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, Karthik SridharanBandits with Feedback Graphs and Switching CostsRaman Arora, Teodor V. Marinov, Mehryar MohriRegularized Gradient BoostingCorinna Cortes, Mehryar Mohri, Dmitry StorcheusLogarithmic Regret for Online ControlNaman Agarwal, Elad Hazan, Karan SinghSampled Softmax with Random Fourier FeaturesAnkit Singh Rawat, Jiecao Chen, Felix Yu, Ananda Theertha Suresh, Sanjiv KumarMultilabel Reductions: What is My Loss Optimising?Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, Sanjiv KumarMetaInit: Initializing Learning by Learning to InitializeYann N. Dauphin, Sam SchoenholzGeneralization Bounds for Neural Networks via Approximate Description LengthAmit Daniely, Elad GranotVariance Reduction of Bipartite Experiments through Correlation ClusteringJean Pouget-Abadie, Kevin Aydin, Warren Schudy, Kay Brodersen, Vahab MirrokniLikelihood Ratios for Out-of-Distribution DetectionJie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji LakshminarayananCan You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset ShiftYaniv Ovadia, Emily Fertig, Jie Jessie Ren, D. Sculley, Josh Dillon, Sebastian Nowozin, Zack Nado, Balaji Lakshminarayanan, Jasper SnoekSurrogate Objectives for Batch Policy Optimization in One-step Decision MakingMinmin Chen, Ramki Gummadi, Chris Harris, Dale SchuurmansGlobally Optimal Learning for Structured Elliptical LossesYoav Wald, Nofar Noy, Gal Elidan, Ami WieselDPPNet: Approximating Determinantal Point Processes with Deep NetworksZelda Mariet, Yaniv Ovadia, Jasper SnoekGraph Normalizing FlowsJenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, Kevin SwerskyWhen Does Label Smoothing Help?Rafael Muller, Simon Kornblith, Geoff HintonOn the Role of Inductive Bias From Simulation and the Transfer to the Real World: a new Disentanglement DatasetMuhammad Waleed Gondal, Manuel Wüthrich, Đorđe Miladinović, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, Stefan BauerOn the Fairness of Disentangled RepresentationsFrancesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, Olivier BachemAre Disentangled Representations Helpful for Abstract Visual Reasoning?Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, Olivier BachemDon’t Blame the ELBO! A Linear VAE Perspective on Posterior CollapseJames Lucas, George Tucker, Roger Grosse, Mohammad NorouziStabilizing Off-Policy Q-Learning via Bootstrapping Error ReductionAviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey LevineOptimizing Generalized Rate Metrics with Game EquilibriumHarikrishna Narasimhan, Andrew Cotter, Maya GuptaOn Making Stochastic Classifiers DeterministicAndrew Cotter, Harikrishna Narasimhan, Maya GuptaDiscrete Flows: Invertible Generative Models of Discrete DataDustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, Ben PooleGraph Agreement Models for Semi-Supervised LearningOtilia Stretcu, Krishnamurthy Viswanathan, Dana Movshovitz-Attias, Emmanouil Platanios, Andrew Tomkins, Sujith RaviA Robust Non-Clairvoyant Dynamic Mechanism for Contextual AuctionsYuan Deng, Sébastien Lahaie, Vahab MirrokniAdversarial Robustness through Local LinearizationChongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy (Dj) Dvijotham, Alhusein Fawzi, Soham De, Robert Stanforth, Pushmeet KohliA Geometric Perspective on Optimal Representations for Reinforcement LearningMarc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, Clare LyleOnline Learning via the Differential Privacy LensJacob Abernethy, Young Hun Jung, Chansoo Lee, Audra McMillan, Ambuj TewariReducing the Variance in Online Optimization by Transporting Past GradientsSébastien M. R. Arnold, Pierre-Antoine Manzagol, Reza Babanezhad, Ioannis Mitliagkas, Nicolas Le RouxUniversality and Individuality in Neural Dynamics Across Large Populations of Recurrent NetworksNiru Maheswaranathan, Alex Williams, Matt Golub, Surya Ganguli, David SussilloReverse Engineering Recurrent Networks for Sentiment Classification Reveals Line Attractor DynamicsNiru Maheswaranathan, Alex H. Williams, Matthew D. Golub, Surya Ganguli, David SussilloStrategizing Against No-Regret LearnersYuan Deng, Jon Schneider, Balasubramanian SivanPrior-Free Dynamic Auctions with Low Regret BuyersYuan Deng, Jon Schneider, Balasubramanian SivanPrivate Stochastic Convex Optimization with Optimal RatesRaef Bassily, Vitaly Feldman, Kunal Talwar, Abhradeep ThakurtaComputational Separations between Sampling and OptimizationKunal TalwarMomentum-Based Variance Reduction in Non-Convex SGDAshok Cutkosky and Francesco OrabonaKernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise AccelerationKwang-Sung Jun, Ashok Cutkosky, Francesco OrabonaFast and Flexible Multi-Task Classification using Conditional Neural Adaptive ProcessesJames Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, Richard E. TurnerIcebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian ModelWenbo Gong, Sebastian Tschiatschek, Richard E. Turner, Sebastian Nowozin, Jose Miguel Hernandez-Lobato, Cheng ZhangMultiview Aggregation for Learning Category-Specific Shape ReconstructionSrinath Sridhar, Davis Rempe, Julien Valentin, Sofien Bouaziz, Leonidas J. GuibasVisualizing and Measuring the Geometry of BERTAndy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, Martin WattenbergLocality-Sensitive Hashing for f-Divergences: Mutual Information Loss and BeyondLin Chen, Hossein Esfandiari, Thomas Fu, Vahab S. MirrokniA Benchmark for Interpretability Methods in Deep Neural NetworksSara Hooker, Dumitru Erhan, Pieter-jan Kindermans, Been KimPractical and Consistent Estimation of f-DivergencesPaul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, Ilya TolstikhinTree-Sliced Variants of Wasserstein DistancesTam Le, Makoto Yamada, Kenji Fukumizu, Marco CuturiGame Design for Eliciting Distinguishable BehaviorFan Yang, Liu Leqi, Yifan Wu, Zachary Lipton, Pradeep Ravikumar, Tom M Mitchell, William CohenDifferentially Private Anonymized HistogramsAnanda Theertha SureshLocally Private Gaussian EstimationMatthew Joseph, Janardhan Kulkarni, Jieming Mao, Zhiwei Steven WuExponential Family Estimation via Adversarial Dynamics EmbeddingBo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, Dale SchuurmansLearning to Predict Without Looking Ahead: World Models Without Forward PredictionC. Daniel Freeman, Luke Metz, David HaAdaptive Density Estimation for Generative ModelsThomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, Jakob VerbeekWeight Agnostic Neural NetworksAdam Gaier, David HaRetrosynthesis Prediction with Conditional Graph Logic NetworkHanjun Dai, Chengtao Li, Connor Coley, Bo Dai, Le SongLarge Scale Structure of Neural Network Loss LandscapesStanislav Fort, Stainslaw JastrzebskiOff-Policy Evaluation via Off-Policy ClassificationAlex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, Sergey LevineDomes to Drones: Self-Supervised Active Triangulation for 3D Human Pose ReconstructionAleksis Pirinen, Erik Gartner, Cristian SminchisescuEnergy-Inspired Models: Learning with Sampler-Induced DistributionsDieterich Lawson, George Tucker, Bo Dai, Rajesh RanganathFrom Deep Learning to Mechanistic Understanding in Neuroscience: The Structure of Retinal PredictionHidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, Surya Ganguli
Language as an Abstraction for Hierarchical Deep Reinforcement LearningYiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea FinnBayesian Layers: A Module for Neural Network UncertaintyDustin Tran, Michael W. Dusenberry, Mark van der Wilk, Danijar HafnerAdaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty EstimatesHugo Penedones, Carlos Riquelme, Damien Vincent, Hartmut Maennel, Timothy Mann, Andre Barreto, Sylvain Gelly, Gergely NeuA Unified Framework for Data Poisoning Attack to Graph-based Semi-Supervised LearningXuanqing Liu, Si Si, Xiaojin Zhu, Yang Li, Cho-Jui HsiehMixMatch: A Holistic Approach to Semi-Supervised LearningDavid Berthelot, Nicholas Carlini, Ian Goodfellow (work done while at Google), Avital Oliver, Nicolas Papernot, Colin RaffelSMILe: Scalable Meta Inverse Reinforcement Learning through Context-Conditional PoliciesSeyed Kamyar Seyed Ghasemipour, Shixiang (Shane) Gu, Richard ZemelLimits of Private Learning with Access to Public DataNoga Alon, Raef Bassily, Shay MoranRegularized Weighted Low Rank ApproximationFrank Ban, David Woodruff, Richard ZhangUnsupervised Curricula for Visual Meta-Reinforcement LearningAllan Jabri, Kyle Hsu, Abhishek Gupta, Benjamin Eysenbach, Sergey Levine, Chelsea FinnSecretary Ranking with Minimal InversionsSepehr Assadi, Eric Balkanski, Renato Paes LemeMixtape: Breaking the Softmax Bottleneck EfficientlyZhilin Yang, Thang Luong, Russ Salakhutdinov, Quoc V. LeBudgeted Reinforcement Learning in Continuous State SpaceNicolas Carrara, Edouard Leurent, Romain Laroche, Tanguy Urvoy, Odalric-Ambrym Maillard, Olivier PietquinFrom Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox OptimizationKrzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao TangFlattening a Hierarchical Clustering through Active LearningFabio Vitale, Anand Rajagopalan, Claudio GentileRobust Attribution RegularizationJiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, Somesh JhaRobustness Verification of Tree-based ModelsHongge Chen, Huan Zhang, Si Si, Yang Li, Duane Boning, Cho-Jui HsiehMeta Architecture SearchAlbert Shaw, Wei Wei, Weiyang Liu, Le Song, Bo DaiContextual Bandits with Cross-LearningSantiago Balseiro, Negin Golrezaei, Mohammad Mahdian, Vahab Mirrokni, Jon SchneiderDynamic Incentive-Aware Learning: Robust Pricing in Contextual AuctionsNegin Golrezaei, Adel Javanmard, Vahab MirrokniOptimizing Generalized Rate Metrics with Three PlayersHarikrishna Narasimhan, Andrew Cotter, Maya GuptaNoise-Tolerant Fair ClassificationAlexandre Louis Lamy, Ziyuan Zhong, Aditya Krishna Menon, Nakul VermaTowards Automatic Concept-based ExplanationsAmirata Ghorbani, James Wexler, James Zou, Been KimLocally Private Learning without Interaction Requires SeparationAmit Daniely, Vitaly FeldmanLearning GANs and Ensembles Using DiscrepancyBen Adlam, Corinna Cortes, Mehryar Mohri, Ningshan ZhangCondConv: Conditionally Parameterized Convolutions for Efficient InferenceBrandon Yang, Gabriel Bender, Quoc V. Le, Jiquan NgiamA Fourier Perspective on Model Robustness in Computer VisionDong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin GilmerRobust Bi-Tempered Logistic Loss Based on Bregman DivergencesEhsan Amid, Manfred K. Warmuth, Rohan Anil, Tomer KorenWhich Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic ModelGuodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, Roger GrosseWide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentJaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey PenningtonAbstract Reasoning with Distracting FeaturesKecheng Zheng, Zheng-Jun Zha, Wei WeiSearch on the Replay Buffer: Bridging Planning and Reinforcement LearningBenjamin Eysenbach, Ruslan Salakhutdinov, Sergey LevineDifferentiable Ranking and Sorting Using Optimal TransportMarco Cuturi, Olivier Teboul, Jean-Philippe VertXLNet: Generalized Autoregressive Pretraining for Language UnderstandingZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. LePrivate Learning Implies Online Learning: An Efficient ReductionAlon Gonen, Elad Hazan, Shay MoranEvaluating Protein Transfer Learning with TAPERoshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, Yun SongTight Dimensionality Reduction for Sketching Low Degree Polynomial KernelsMichela Meister, Tamas Sarlos, David P. WoodruffNo Pressure! Addressing the Problem of Local Minima in Manifold Learning AlgorithmsMax VladymyrovSubspace Detours: Building Transport Plans that are Optimal on Subspace ProjectionsBoris Muzellec, Marco CuturiOn the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement DatasetMuhammad Waleed Gondal, Manuel Wüthrich, Ðorde Miladinovíc, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, Stefan BauerStacked Capsule AutoencodersAdam R. Kosiorek, Sara Sabour, Yee Whye Teh, Geoffrey E. HintonWasserstein Dependency Measure for Representation LearningSherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, Pierre SermanetSampling Sketches for Concave Sublinear Functions of FrequenciesEdith Cohen, Ofir GeriHamiltonian Neural NetworksSam Greydanus, Misko Dzamba, Jason YosinskiComputational Mirrors: Blind Inverse Light Transport by Deep Matrix FactorizationMiika Aittala, Prafull Sharma, Lukas Murmann, Adam B. Yedidia, Gregory W. Wornell, William T. Freeman, Frédo DurandTransfusion: Understanding Transfer Learning for Medical ImagingMaithra Raghu, Chiyuan Zhang, Jon Kleinberg, Samy BengioDifferentially Private Covariance EstimationKareem Amin, Travis Dick, Alex Kulesza, Andres Munoz, Sergei VassilvitskiiLearning Transferable Graph ExplorationHanjun Dai, Yujia Li, Chenglong Wang, Rishabh Singh, Po-Sen Huang, Pushmeet KohliNeural Attribution for Semantic Bug-Localization in Student ProgramsRahul Gupta, Aditya Kanade, Shirish ShevadePyTorch: An Imperative Style, High-Performance Deep Learning LibraryAdam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith ChintalaBreaking the Glass Ceiling for Embedding-Based Classifiers for Large Output SpacesChuan Guo, Ali Mousavi, Xiang Wu, Daniel Holtmann-Rice, Satyen Kale, Sashank Reddi, Sanjiv KumarEfficient Rematerialization for Deep NetworksRavi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, Joshua R. WangWorkshops3rd Conversational AI: Today's Practice and Tomorrow's PotentialBill ByrneAI for Humanitarian Assistance and Disaster Response WorkshopYossi MatiasBayesian Deep LearningKevin P MurphyBeyond First Order Methods in Machine Learning SystemsElad HazanBiological and Artificial Reinforcement LearningIgor MordatchContext and Compositionality in Biological and Artificial Neural SystemsKenton LeeDeep Reinforcement LearningChelsea FinnDocument IntelligenceTania Bedrax WeissFederated Learning for Data Privacy and ConfidentialityJakub KonečnýBrendan McMahanFrançoise Beaufays, Daniel RamageGraph Representation LearningRianne van den BergHuman-Centric Machine LearningBeen KimInformation Theory and Machine LearningBen PooleAlex AlemiKR2ML - Knowledge Representation and Reasoning Meets Machine LearningWilliam CohenLearning Meaningful Representations of LifeJasper SnoekAlexander WiltschkoLearning Transferable SkillsDavid HaMachine Learning for Creativity and DesignAdam RobertsJesse EngelMachine Learning for Health (ML4H): What Makes Machine Learning in Medicine Different?Lily Peng, Alan Karthikesalingam, Dale WebsterMachine Learning and the Physical SciencesYasaman Bahri, Samual SchoenholzML for SystemsMilad HashemiKevin SwerskyAzalia MirhoseiniAnna GoldieJeff DeanOptimal Transport for Machine LearningMarco CuturiThe Optimization Foundations of Reinforcement LearningBo DaiNicolas Le RouxLihong LiDale SchuurmansPrivacy in Machine LearningBrendan McMahanProgram Transformations for MLPascal LamblinAlexander WiltschkoBart van MerrienboerEmily FertigSkye Wanderman-MilneReal Neurons & Hidden Units: Future Directions at the Intersection of Neuroscience and Artificial IntelligenceDavid SussilloRobot Learning: Control and Interaction in the Real WorldStefan SchaalSafety and Robustness in Decision MakingYinlam ChowScience Meets Engineering of Deep LearningYasaman Bahri, Surya Ganguli‎, Been Kim, Surya GanguliSets and PartitionsManzil ZaheerAndrew McCallumAmr AhmedTackling Climate Change with MLJohn PlattJeff DeanVisually Grounded Interaction and LanguageJason BaldridgeWorkshop on Machine Learning with GuaranteesMehryar MohriTutorialsRepresentation Learning and FairnessMoustapha Cisse, Sanmi Koyejo

Blog