Transformer models consistently obtain state-of-the-art results in computer vision tasks, including object detection and video classification. In contrast to standard convolutional approaches that process images pixel-by-pixel, the Vision Transformers (ViT) treat an image as a sequence of patch tokens (i.e., a smaller part, or “patch”, of an image made up of multiple pixels). This means that at every layer, a ViT model recombines and processes patch tokens based on relations between each pair of tokens, using multi-head self-attention. In doing so, ViT models have the capability to construct a global representation of the entire image.
At the input-level, the tokens are formed by uniformly splitting the image into multiple segments, e.g., splitting an image that is 512 by 512 pixels into patches that are 16 by 16 pixels. At the intermediate levels, the outputs from the previous layer become the tokens for the next layer. In the case of videos, video ‘tubelets’ such as 16x16x2 video segments (16x16 images over 2 frames) become tokens. The quality and quantity of the visual tokens decide the overall quality of the Vision Transformer.
The main challenge in many Vision Transformer architectures is that they often require too many tokens to obtain reasonable results. Even with 16x16 patch tokenization, for instance, a single 512x512 image corresponds to 1024 tokens. For videos with multiple frames, that results in tens of thousands of tokens needing to be processed at every layer. Considering that the Transformer computation increases quadratically with the number of tokens, this can often make Transformers intractable for larger images and longer videos. This leads to the question: is it really necessary to process that many tokens at every layer?
In “TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?”, an earlier version of which is presented at NeurIPS 2021, we show that adaptively generating a smaller number of tokens, rather than always relying on tokens formed by uniform splitting, enables Vision Transformers to run much faster and perform better. TokenLearner is a learnable module that takes an image-like tensor (i.e., input) and generates a small set of tokens. This module could be placed at various different locations within the model of interest, significantly reducing the number of tokens to be handled in all subsequent layers. The experiments demonstrate that having TokenLearner saves memory and computation by half or more without damaging classification performance, and because of its ability to adapt to inputs, it even increases the accuracy.
The TokenLearner We implement TokenLearner using a straightforward spatial attention approach. In order to generate each learned token, we compute a spatial attention map highlighting regions-of-importance (using convolutional layers or MLPs). Such a spatial attention map is then applied to the input to weight each region differently (and discard unnecessary regions), and the result is spatially pooled to generate the final learned tokens. This is repeated multiple times in parallel, resulting in a few (~10) tokens out of the original input. This can also be viewed as performing a soft-selection of the pixels based on the weight values, followed by global average pooling. Note that the functions to compute the attention maps are governed by different sets of learnable parameters, and are trained in an end-to-end fashion. This allows the attention functions to be optimized in capturing different spatial information in the input. The figure below illustrates the process.
As a result, instead of processing fixed, uniformly tokenized inputs, TokenLearner enables models to process a smaller number of tokens that are relevant to the specific recognition task. That is, (1) we enable adaptive tokenization so that the tokens can be dynamically selected conditioned on the input, and (2) this effectively reduces the total number of tokens, greatly reducing the computation performed by the network. These dynamically and adaptively generated tokens can be used in standard transformer architectures such as ViT for images and ViViT for videos.
Where to Place TokenLearner After building the TokenLearner module, we had to determine where to place it. We first tried placing it at different locations within the standard ViT architecture with 224x224 images. The number of tokens TokenLearner generated was 8 and 16, much less than 196 or 576 tokens the standard ViTs use. The below figure shows ImageNet few-shot classification accuracies and FLOPS of the models with TokenLearner inserted at various relative locations within ViT B/16, which is the base model with 12 attention layers operating on 16x16 patch tokens.
We found that inserting TokenLearner after the initial quarter of the network (at 1/4) achieves almost identical accuracies as the baseline, while reducing the computation to less than a third of the baseline. In addition, placing TokenLearner at the later layer (after 3/4 of the network) achieves even better performance compared to not using TokenLearner while performing faster, thanks to its adaptiveness. Due to the large difference between the number of tokens before and after TokenLearner (e.g., 196 before and 8 after), the relative computation of the transformers after the TokenLearner module becomes almost negligible.
Comparing Against ViTs We compared the standard ViT models with TokenLearner against those without it while following the same setting on ImageNet few-shot transfer. TokenLearner was placed in the middle of each ViT model at various locations such as at 1/2 and at 3/4. The below figure shows the performance/computation trade-off of the models with and without TokenLearner.
We also inserted TokenLearner within larger ViT models, and compared them against the giant ViT G/14 model. Here, we applied TokenLearner to ViT L/10 and L/8, which are the ViT models with 24 attention layers taking 10x10 (or 8x8) patches as initial tokens. The below figure shows that despite using many fewer parameters and less computation, TokenLearner performs comparably to the giant G/14 model with 48 layers.
High-Performing Video Models Video understanding is one of the key challenges in computer vision, so we evaluated TokenLearner on multiple video classification datasets. This was done by adding TokenLearner into Video Vision Transformers (ViViT), which can be thought of as a spatio-temporal version of ViT. TokenLearner learned 8 (or 16) tokens per timestep.
When combined with ViViT, TokenLearner obtains state-of-the-art (SOTA) performance on multiple popular video benchmarks, including Kinetics-400, Kinetics-600, Charades, and AViD, outperforming the previous Transformer models on Kinetics-400 and Kinetics-600 as well as previous CNN models on Charades and AViD.
Conclusion While Vision Transformers serve as powerful models for computer vision, a large number of tokens and their associated computation amount have been a bottleneck for their application to larger images and longer videos. In this project, we illustrate that retaining such a large number of tokens and fully processing them over the entire set of layers is not necessary. Further, we demonstrate that by learning a module that extracts tokens adaptively based on the input image allows attaining even better performance while saving compute. The proposed TokenLearner was particularly effective in video representation learning tasks, which we confirmed with multiple public datasets. A preprint of our work as well as code are publicly available.
Acknowledgement We thank our co-authors: AJ Piergiovanni, Mostafa Dehghani, and Anelia Angelova. We also thank the Robotics at Google team members for the motivating discussions.
This week marks the beginning of the 35th annual Conference on Neural Information Processing Systems (NeurIPS 2021), the biggest machine learning conference of the year. NeurIPS 2021 will be held virtually and includes invited talks, demonstrations and presentations of some of the latest in machine learning research. This year, NeurIPS also announced a new Datasets and Benchmarks track, which will include publications, talks, posters, and discussions related to this research area.
Google will have a strong presence with more than 170 accepted papers, additionally contributing to and learning from the broader academic research community via talks, posters, workshops, and tutorials. You can learn more about our work being presented in the list below (Google affiliations highlighted in bold).
Organizing Committee
Communications Co-Chair: Emily Denton Program Co-Chair: Yann Dauphin Workshop Co-Chair: Sanmi Koyejo
Senior Area Chairs: Alekh Agarwal, Amir Globerson, Been Kim, Charles Sutton, Claudio Gentile, Corinna Cortes, Dale Schuurmans, David Duvenaud, Elad Hazan, Hugo Larochelle, Jean-Philippe Vert, Kevin Murphy, Marco Cuturi, Mehryar Mohri, Mohammad Ghavamzadeh, Samory Kpotufe, Sanjiv Kumar, Satyen Kale, Sergey Levine, Tara N. Sainath, Yishay Mansour
Area Chairs: Abhishek Kumar, Abhradeep Guha Thakurta, Alex Kulesza, Alexander A. Alemi, Alexander T. Toshev, Amin Karbasi, Amit Daniely, Ananda Theertha Suresh, Ankit Singh Rawat, Ashok Cutkosky, Badih Ghazi, Balaji Lakshminarayanan, Ben Poole, Bo Dai, Boqing Gong, Chelsea Finn, Chiyuan Zhang, Christian Szegedy, Cordelia Schmid, Craig Boutilier, Cyrus Rashtchian, D. Sculley, Daniel Keysers, David Ha, Denny Zhou, Dilip Krishnan, Dumitru Erhan, Dustin Tran, Ekin Dogus Cubuk, Fabian Pedregosa, George Tucker, Hanie Sedghi, Hanjun Dai, Heinrich Jiang, Hossein Mobahi, Izhak Shafran, Jaehoon Lee, Jascha Sohl-Dickstein, Jasper Snoek, Jeffrey Pennington, Jelani Nelson, Jieming Mao, Justin Gilmer, Karol Hausman, Karthik Sridharan, Kevin Swersky, Maithra Raghu, Mario Lucic, Mathieu Blondel, Matt Kusner, Matthew Johnson, Matthieu Geist, Ming-Hsuan Yang, Mohammad Mahdian, Mohammad Norouzi, Nal Kalchbrenner, Naman Agarwal, Nicholas Carlini, Nicolas Papernot, Olivier Bachem, Olivier Pietquin, Paul Duetting, Praneeth Netrapalli, Pranjal Awasthi, Prateek Jain, Quentin Berthet, Renato Paes Leme, Richard Nock, Rif A. Saurous, Rose Yu, Roy Frostig, Samuel Stern Schoenholz, Sashank J. Reddi, Sercan O. Arik, Sergei Vassilvitskii, Sergey Ioffe, Shay Moran, Silvio Lattanzi, Simon Kornblith, Srinadh Bhojanapalli, Thang Luong, Thomas Steinke, Tim Salimans, Tomas Pfister, Tomer Koren, Uri Stemmer, Vahab Mirrokni, Vikas Sindhwani, Vincent Dumoulin, Virginia Smith, Vladimir Braverman, W. Ronny Huang, Wen Sun, Yang Li, Yasin Abbasi-Yadkori, Yinlam Chow,Yujia Li, Yunhe Wang, Zoltán Szabó
NeurIPS Foundation Board 2021: Michael Mozer, Corinna Cortes, Hugo Larochelle, John C. Platt, Fernando Pereira
Test of Time Award
Online Learning for Latent Dirichlet Allocation Matthew D. Hoffman†, David M. Blei, Francis Bach
Publications
Deep Reinforcement Learning at the Edge of the Statistical Precipice (see blog post) Outstanding Paper Award Recipient Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare
A Separation Result Between Data-Oblivious and Data-Aware Poisoning Attacks Samuel Deng, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Abhradeep Guha Thakurta
Adversarial Robustness of Streaming Algorithms Through Importance Sampling Vladimir Braverman, Avinatan Hassidim, Yossi Matias, Mariano Schain, Sandeep Silwal, Samson Zhou
Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery Mugallodi Rakesh, Jogendra Nath Kundu, Varun Jampani, R. Venkatesh Babu
Attention Bottlenecks for Multimodal Fusion Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun
Autonomous Reinforcement Learning via Subgoal Curricula Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn
Calibration and Consistency of Adversarial Surrogate Losses Pranjal Awasthi, Natalie S. Frank, Anqi Mao, Mehryar Mohri, Yutao Zhong
Compressive Visual Representations Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, Ian Fischer
Counterfactual Invariance to Spurious Correlations in Text Classification Victor Veitch, Alexander D'Amour, Steve Yadlowsky, Jacob Eisenstein
Deep Learning Through the Lens of Example Difficulty Robert J.N. Baldock, Hartmut Maennel, Behnam Neyshabur
Deep Neural Networks as Point Estimates for Deep Gaussian Processes Vinent Dutordoir, James Hensman, Mark van der Wilk, Carl Henrik Ek, Zoubin Ghahramani, Nicolas Durrande
Delayed Gradient Averaging: Tolerate the Communication Latency for Federated Learning Ligeng Zhu, Hongzhou Lin, Yao Lu, Yujun Lin, Song Han
Discrete-Valued Neural Communication Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer, Yoshua Bengio
Do Vision Transformers See Like Convolutional Neural Networks? Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy
Dueling Bandits with Team Comparisons Lee Cohen, Ulrike Schmidt-Kraepelin, Yishay Mansour
End-to-End Multi-Modal Video Temporal Grounding Yi-Wen Chen, Yi-Hsuan Tsai, Ming-Hsuan Yang
Environment Generation for Zero-Shot Compositional Reinforcement Learning Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, Aleksandra Faust
H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion Hongyi Xu, Thiemo Alldieck, Cristian Sminchisescu
Improving Calibration Through the Relationship with Adversarial Robustness Yao Qin, Xuezhl Wang, Alex Beutel, Ed Chi
Learning Generalized Gumbel-Max Causal Mechanisms Guy Lorberbom, Daniel D. Johnson, Chris J. Maddison, Daniel Tarlow, Tamir Hazan
MICo: Improved Representations via Sampling-Based State Similarity for Markov Decision Processes Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, Mark Rowland
Near-Optimal Lower Bounds For Convex Optimization For All Orders of Smoothness Ankit Garg, Robin Kothari, Praneeth Netrapalli, Suhail Sherif
Neural Circuit Synthesis from Specification Patterns Frederik Schmitt, Christopher Hahn, Markus N. Rabe, Bernd Finkbeiner
Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation Jogendra Nath Kundu, Siddharth Seth, Anirudh Jamkhandi, Pradyumna YM, Varun Jampani, Anirban Chakraborty, R. Venkatesh Babu
Object-Aware Contrastive Learning for Debiased Scene Representation Sangwoo Mo, Hyunwoo Kang, Kihyuk Soh, Chun-Liang Li, Jinwoo Shin
On Density Estimation with Diffusion Models Diederik P. Kingma, Tim Salimans, Ben Poole, Jonathan Ho
On Margin-Based Cluster Recovery with Oracle Queries Marco Bressan, Nicolo Cesa-Bianchi, Silvio Lattanzi, Andrea Paudice
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation Tai-Yu Pan, Cheng Zhang, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, Wei-Lun Chao
Parallelizing Thompson Sampling Amin Karbasi, Vahab Mirrokni, Mohammad Shadravan
Reverse-Complement Equivariant Networks for DNA Sequences Vincent Mallet, Jean-Philippe Vert
Revisiting ResNets: Improved Training and Scaling Strategies Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph
Revisiting the Calibration of Modern Neural Networks Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic
Scaling Vision with Sparse Mixture of Experts Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, Neil Houlsby
SE(3)-Equivariant Prediction of Molecular Wavefunctions and Electronic Densities Oliver Thorsten Unke, Mihail Bogojeski, Michael Gastegger, Mario Geiger, Tess Smidt, Klaus Robert Muller
Stateful ODE-Nets Using Basis Function Expansions Alejandro Francisco Queiruga, N. Benjamin Erichson, Liam Hodgkinson, Michael W. Mahoney
Statistically and Computationally Efficient Linear Meta-Representation Learning Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, Sewoong Oh
Streaming Belief Propagation for Community Detection Yuchen Wu, Jakab Tardos, Mohammad Hossein Bateni, André Linhares, Filipe Miguel Gonçalves de Almeida, Andrea Montanari, Ashkan Norouzi-Fard
Synthetic Design: An Optimization Approach to Experimental Design with Synthetic Controls Nick Doudchenko, Khashayar Khosravi, Jean Pouget-Abadie, Sebastien Lahaie, Miles Lubin, Vahab Mirrokni, Jann Spiess, Guido Imbens
The Difficulty of Passive Learning in Deep Reinforcement Learning George Ostrovski, Pablo Samuel Castro, Will Dabney
The Pareto Frontier of Model Selection for General Contextual Bandits Teodor Marinov, Julian Zimmert
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, Boqing Gong
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-Based Deep Reinforcement Learning Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, Shixiang Gu
Conservative Data Sharing for Multi-Task Offline Reinforcement Learning Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, Chelsea Finn
Does Knowledge Distillation Really Work? Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, Andrew Gordon Wilson
Exponential Graph is Provably Efficient for Decentralized Deep Training Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, Wotao Yin
Faster Matchings via Learned Duals Michael Dinitz, Sungjin Im, Thomas Lavastida, Benjamin Moseley, Sergei Vassilvitskii
Improved Transformer for High-Resolution GANs Long Zhao, Zizhao Zhang, Ting Chen, Dimitris N. Metaxas, Han Zhang
Near-Optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems Prateek Jain, Suhas S. Kowshik, Dheeraj Mysore Nagaraj, Praneeth Netrapalli
Nearly Horizon-Free Offline Reinforcement Learning Tongzheng Ren, Jialian Li, Bo Dai, Simon S. Du, Sujay Sanghavi
Overparameterization Improves Robustness to Covariate Shift in High Dimensions Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
Pay Attention to MLPs Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le
PLUR: A Unifying, Graph-Based View of Program Learning, Understanding, and Repair Zimin Chen*, Vincent Josua Hellendoorn*, Pascal Lamblin, Petros Maniatis, Pierre-Antoine Manzagol, Daniel Tarlow, Subhodeep Moitra
Prior-Independent Dynamic Auctions for a Value-Maximizing Buyer Yuan Deng, Hanrui Zhang
Remember What You Want to Forget: Algorithms for Machine Unlearning Ayush Sekhari, Jayadev Acharya, Gautam Kamath, Ananda Theertha Suresh
Reverse Engineering Learned Optimizers Reveals Known and Novel Mechanisms Niru Maheswaranathan*, David Sussillo*, Luke Metz, Ruoxi Sun, Jascha Sohl-Dickstein
Revisiting 3D Object Detection From an Egocentric Perspective Boyang Deng, Charles R. Qi, Mahyar Najibi, Thomas Funkhouser, Yin Zhou, Dragomir Anguelov
Robust Auction Design in the Auto-Bidding World Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, Song Zuo
Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data Qi Zhu, Natalia Ponomareva, Jiawei Han, Bryan Perozzi
Understanding How Encoder-Decoder Architectures Attend Kyle Aitken, Vinay V. Ramasesh, Yuan Cao, Niru Maheswaranathan
Understanding the Effect of Stochasticity in Policy Optimization Jincheng Mei, Bo Dai, Chenjun Xiao, Csaba Szepesvari, Dale Schuurmans
Accurately Solving Rod Dynamics with Graph Learning Han Shao, Tassilo Kugelstadt, Torsten Hädrich, Wojtek Palubicki, Jan Bender, Sören Pirk, Dominik L. Michels
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W. Ronny Huang, Tom Goldstein
Learnability of Linear Thresholds from Label Proportions Rishi Saket
MLP-Mixer: An All-MLP Architecture for Vision Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy
Neural Additive Models: Interpretable Machine Learning with Neural Nets Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, Geoffrey Hinton
Neural Production Systems Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, Yoshua Bengio
Physics-Aware Downsampling with Deep Learning for Scalable Flood Modeling Niv Giladi, Zvika Ben-Haim, Sella Nevo, Yossi Matias, Daniel Soudry
Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects Denys Rozumnyi, Martin R. Oswald, Vittorio Ferrari, Marc Pollefeys
What Matters for Adversarial Imitation Learning? Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, Marcin Andrychowicz
A Convergence Analysis of Gradient Descent on Graph Neural Networks Pranjal Awasthi, Abhimanyu Das, Sreenivas Gollapudi
A Geometric Analysis of Neural Collapse with Unconstrained Features Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, Qing Qu
Agnostic Reinforcement Learning with Low-Rank MDPs and Rich Observations Christoph Dann, Yishay Mansour, Mehryar Mohri, Ayush Sekhari, Karthik Sridharan
Controlled Text Generation as Continuous Optimization with Multiple Constraints Sachin Kumar, Eric Malmi, Aliaksei Severyn, Yulia Tsvetkov
Coupled Gradient Estimators for Discrete Latent Variables Zhe Dong, Andriy Mnih, George Tucker
Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-Training Ensembles Jiefeng Chen*, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, Somesh Jha
Neural Active Learning with Performance Guarantees Zhilei Wang, Pranjal Awasthi, Christoph Dann, Ayush Sekhari, Claudio Gentile
Optimal Sketching for Trace Estimation Shuli Jiang, Hai Pham, David Woodruff, Qiuyi (Richard) Zhang
Representing Long-Range Context for Graph Neural Networks with Global Attention Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, Ion Stoica
Scaling Up Exact Neural Network Compression by ReLU Stability Thiago Serra, Xin Yu, Abhinav Kumar, Srikumar Ramalingam
Soft Calibration Objectives for Neural Networks Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael Curtis Mozer, Rebecca Roelofs
Sub-Linear Memory: How to Make Performers SLiM Valerii Likhosherstov, Krzysztof Choromanski, Jared Davis, Xingyou Song, Adrian Weller
A New Theoretical Framework for Fast and Accurate Online Decision-Making Nicolò Cesa-Bianchi, Tommaso Cesari, Yishay Mansour, Vianney Perchet
Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, Radu Soricut
Differentially Private Multi-Armed Bandits in the Shuffle Model Jay Tenenbaum, Haim Kaplan, Yishay Mansour, Uri Stemmer
Efficient and Local Parallel Random Walks Michael Kapralov, Silvio Lattanzi, Navid Nouri, Jakab Tardos
Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss Michael Louis Iuzzolino, Michael Curtis Mozer, Samy Bengio*
It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems Regev Cohen, Yochai Blau, Daniel Freedman, Ehud Rivlin
Learning to Combine Per-Example Solutions for Neural Program Synthesis Disha Shrivastava, Hugo Larochelle, Daniel Tarlow
LLC: Accurate, Multi-purpose Learnt Low-Dimensional Binary Codes Aditya Kusupati, Matthew Wallingford, Vivek Ramanujan, Raghav Somani, Jae Sung Park, Krishna Pillutla, Prateek Jain, Sham Kakade, Ali Farhadi
There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning (see blog post)
Nathan Grinsztajn, Johan Ferret, Olivier Pietquin, Philippe Preux, Matthieu Geist
A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models Ibrahim Alabdulmohsin, Mario Lucic
Adaptive Sampling for Minimax Fair Classification Shubhanshu Shekhar, Greg Fields, Mohammad Ghavamzadeh, Tara Javidi
Asynchronous Stochastic Optimization Robust to Arbitrary Delays Alon Cohen, Amit Daniely, Yoel Drori, Tomer Koren, Mariano Schain
Boosting with Multiple Sources Corinna Cortes, Mehryar Mohri, Dmitry Storcheus, Ananda Theertha Suresh
Breaking the Centralized Barrier for Cross-Device Federated Learning Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stitch, Ananda Theertha Sureshi
Canonical Capsules: Self-Supervised Capsules in Canonical Pose Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, Kwang Moo Yi
Contextual Recommendations and Low-Regret Cutting-Plane Algorithms Sreenivas Gollapudi, Guru Guruganesh, Kostas Kollias, Pasi Manurangsi, Renato Paes Leme, Jon Schneider
Decision Transformer: Reinforcement Learning via Sequence Modeling Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee|Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch
Deep Learning on a Data Diet: Finding Important Examples Early in Training Mansheej Paul, Surya Ganguli, Gintare Karolina Dziugaite
Deep Learning with Label Differential Privacy Badih Ghazi, Noah Golowich*, Ravi Kumar, Pasin Manurangsi, Chiyuan Zhang
Efficient Training of Retrieval Models Using Negative Cache Erik Lindgren, Sashank J. Reddi, Ruiqi Guo, Sanjiv Kumar
Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, Ming-Hsuan Yang
Federated Reconstruction: Partially Local Federated Learning Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, Keith Rush, Sushant Prakash
Framing RNN as a Kernel Method: A Neural ODE Approach Adeline Fermanian, Pierre Marion, Jean-Philippe Vert, Gérard Biau
Learning Semantic Representations to Verify Hardware Designs Shobha Vasudevan, Wenjie Jiang, David Bieber, Rishabh Singh, Hamid Shojaei, C. Richard Ho, Charles Sutton
Learning with User-Level Privacy Daniel Asher Nathan Levy*, Ziteng Sun*, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, Ananda Theertha Suresh
Logarithmic Regret from Sublinear Hints Aditya Bhaskara, Ashok Cutkosky, Ravi Kumar, Manish Purohit
Margin-Independent Online Multiclass Learning via Convex Geometry Guru Guruganesh, Allen Liu, Jon Schneider, Joshua Ruizhi Wang
Multiclass Boosting and the Cost of Weak Learning Nataly Brukhim, Elad Hazan, Shay Moran, Indraneel Mukherjee, Robert E. Schapire
Neural-PIL: Neural Pre-integrated Lighting for Reflectance Decomposition Mark Boss, Varun Jampani, Raphael Braun, Ce Liu*, Jonathan T. Barron, Hendrik Lensch
Never Go Full Batch (in Stochastic Convex Optimization) Idan Amir, Yair Carmon, Tomer Koren, Roi Livni
On Large-Cohort Training for Federated Learning Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, Virginia Smith
On the Sample Complexity of Privately Learning Axis-Aligned Rectangles Menachem Sadigurschi, Uri Stemmer
Online Control of Unknown Time-Varying Dynamical Systems Edgar Minasyan, Paula Gradu, Max Simchowitz, Elad Hazan
Online Knapsack with Frequency Predictions Sungjin Im, Ravi Kumar,Mahshid Montazer Qaem, Manish Purohit
Optimal Rates for Random Order Online Optimization Uri Sherman, Tomer Koren, Yishay Mansour
Oracle-Efficient Regret Minimization in Factored MDPs with Unknown Structure Aviv Rosenberg, Yishay Mansour
Practical Large-Scale Linear Programming Using Primal-Dual Hybrid Gradient David Applegate, Mateo Díaz*, Oliver Hinder, Haihao Lu*, Miles Lubin, Brendan O'Donoghue, Warren Schudy
Private and Non-Private Uniformity Testing for Ranking Data Robert Istvan Busa-Fekete, Dimitris Fotakis, Manolis Zampetakis
Privately Learning Subspaces Vikrant Singhal, Thomas Steinke
Provable Representation Learning for Imitation with Contrastive Fourier Features Ofir Nachum, Mengjiao Yang
Safe Reinforcement Learning with Natural Language Constraints Tsung-Yen Yang, Michael Hu, Yinlam Chow, Peter J. Ramadge, Karthik Narasimhan
Searching for Efficient Transformers for Language Modeling David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le
SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression Steve Yadlowsky, Taedong Yun, Cory McLean, Alexander D'Amour
Streaming Linear System Identification with Reverse Experience Replay Prateek Jain, Suhas S. Kowshik, Dheeraj Mysore Nagaraj, Praneeth Netrapalli
The Skellam Mechanism for Differentially Private Federated Learning Naman Agarwal, Peter Kairouz, Ziyu Liu*
TokenLearner: Adaptive Space-Time Tokenization for Videos Michael S. Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, Anelia Angelova
Towards Best-of-All-Worlds Online Learning with Feedback Graphs Liad Erez, Tomer Koren
Training Over-Parameterized Models with Non-decomposable Objectives Harikrishna Narasimhan, Aditya Krishna Menon
Twice Regularized MDPs and the Equivalence Between Robustness and Regularization Esther Derman, Matthieu Geist, Shie Mannor
Unsupervised Learning of Compositional Energy Concepts Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, Igor Mordatch
User-Level Differentially Private Learning via Correlated Sampling Badih Ghazi, Ravi Kumar, Pasin Manurangsi
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu*, Deva Ramanan
A Minimalist Approach to Offline Reinforcement Learning Scott Fujimoto, Shixiang Gu
A Unified View of cGANs With and Without Classifiers Si-An Chen, Chun-Liang Li, Hsuan-Tien Lin
CoAtNet: Marrying Convolution and Attention for All Data Sizes (see blog post) Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan
Combiner: Full Attention Transformer with Sparse Computation Cost Hongyu Ren*, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, Bo Dai
Contrastively Disentangled Sequential Variational Autoencoder Junwen Bai, Weiran Wang, Carla P. Gomes
Controlling Neural Networks with Rule Representations Sungyong Seo, Sercan O. Arik, Jinsung Yoon, Xiang Zhang, Kihyuk Sohn, Tomas Pfister
Dataset Distillation with Infinitely Wide Convolutional Networks Timothy Nguyen*, Roman Novak, Lechao Xiao, Jaehoon Lee
Deep Synoptic Monte-Carlo Planning in Reconnaissance Blind Chess Gregory Clark
Differentially Private Learning with Adaptive Clipping Galen Andrew, Om Thakkar, Swaroop Ramaswamy, Hugh Brendan McMahan
Differentially Private Model Personalization Prateek Jain, Keith Rush, Adam Smith, Shuang Song, Abhradeep Thakurta
Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations Pranjal Awasthi, Alex Tang, Aravindan Vijayaraghavan
Efficiently Identifying Task Groupings for Multi-Task Learning Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, Chelsea Finn
Generalized Shape Metrics on Neural Representations Alex H. Williams, Erin Kunz, Simon Kornblith, Scott Linderman
High-Probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails Ashok Cutkosky, Harsh Mehta
Identity Testing for Mallows Model Róbert Busa-Fekete, Dimitris Fotakis, Balázs Szörényi, Manolis Zampetakis
Learnable Fourier Features for Multi-dimensional Spatial Positional Encoding Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, Samy Bengio*
Learning to Select Exogenous Events for Marked Temporal Point Process Ping Zhang, Rishabh K. Iyer, Ashish V. Tendulkar, Gaurav Aggarwal, Abir De
Meta-learning to Improve Pre-training Aniruddh Raghu, Jonathan Peter Lorraine, Simon Kornblith, Matthew B.A. McDermott, David Duvenaud
Pointwise Bounds for Distribution Estimation Under Communication Constraints Wei-Ning Chen, Peter Kairouz, Ayfer Özgür
REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People Under Weak Supervision Mihai Fieraru, Mihai Zanfir, Teodor Alexandru Szente, Eduard Gabriel Bazavan, Vlad Olaru, Cristian Sminchisescu
Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification Benjamin Eysenbach, Sergey Levine, Ruslan Salakhutdinov
Revealing and Protecting Labels in Distributed Training Trung Dang, Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Peter Chin, Françoise Beaufays
Robust Predictable Control Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine
Robust Visual Reasoning via Language Guided Neural Module Networks Arjun Reddy Akula, Varun Jampani, Soravit Changpinyo, Song-Chun Zhu
Towards Understanding Retrosynthesis by Energy-Based Models Ruoxi Sun, Hanjun Dai, Li Li, Steven Kearnes, Bo Dai
Exploring the Limits of Out-of-Distribution Detection Stanislav Fort, Jie Ren, Balaji Lakshminarayanan
Minimax Regret for Stochastic Shortest Path Alon Cohen, Yonathan Efroni, Yishay Mansour, Aviv Rosenberg
No Regrets for Learning the Prior in Bandits Soumya Basu, Branislav Kveton, Manzil Zaheer, Csaba Szepesvari
Structured Denoising Diffusion Models in Discrete State-Spaces Jacob Austin, Daniel D. Johnsonv, Jonathan Ho, Daniel Tarlow, Rianne van den Berg
The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning (see blog post) Yujin Tang, David Ha
On the Existence of The Adversarial Bayes Classifier Pranjal Awasthi, Natalie Frank, Mehyrar Mohri
Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learning Christopher Dann, Teodor Vanislavov Marinov, Mehryar Mohri, Julian Zimmert
A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learning Christopher Dann, Mehryar Mohri, Tong Zhang, Julian Zimmert
Datasets & Benchmarks Accepted Papers
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research Bernard Koch, Emily Denton, Alex Hanna, Jacob G. Foster Datasets & Benchmarks Best Paper
Constructing a Visual Dataset to Study the Effects of Spatial Apartheid in South Africa Raesetje Sefala, Timnit Gebru, Luzango Mfupe, Nyalleng Moorosi
AI and the Everything in the Whole Wide World Benchmark Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, Alex Hannah
A Unified Few-Shot Classification Benchmark to Compare Transfer and Meta Learning Approaches Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, Hugo Larochelle
The Neural MMO Platform for Massively Multi-agent Research Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, Phillip Isola
Systematic Evaluation of Causal Discovery in Visual Model-Based Reinforcement Learning Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajole, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, Christopher Pal
STEP: Segmenting and Tracking Every Pixel Mark Weber, Jun Xie, Maxwell Collins, Yukun Zhu, Paul Voigtlaender, Hartwig Adam, Bradley Green, Andreas Geiger, Bastian Leibe, Daneil Cremers, Aljosa Osep, Laura Leal-Taixe, Liang-Chieh Chen
Artsheets for Art Datasets Ramya Srinivisan, Emily Denton, Jordan Famularo, Negar Rostamzadeh, Fernando Diaz, Beth Coleman
SynthBio: A Case in Human–AI Collaborative Curation of Text Datasets Ann Yuan, Daphne Ippolito, Vitaly Niolaev, Chris Callison-Burch, Andy Coenen, Sebastian Gehrmann
Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael W. Dusenberry, Ghassen Jerfel, Dustin Tran, Yarin Gal
Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation (see blog post) C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, Olivier Bachem
MLPerf Tiny Benchmark Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish Thakker, Antonio Torrini, Peter Warden, Jay Cordaro, Giuseppe Di Guglielmo, Javier Duarte, Stephen Gibellini, Videet Parekh, Honson Tran, Nhan Tran, Niu Wenxu, Xu Xuesong
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets Simon Mille, Kaustubh D. Dhole, Saad Mahamood, Laura Perez-Beltrachini, Varun Gangal, Mihir Kale, Emiel van Miltenburg, Sebastian Gehrmann
An Empirical Investigation of Representation Learning for Imitation Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huei Lee, Neel Alex, Steven Wang, Ping Luo, Stuart Russell, Pieter Abbeel, Rohin Shah
Multilingual Spoken Words Corpus Mark Mazumder, Sharad Chitlangia, Colby Banbury, Yiping Kang, Juan Manuel Ciro, Keith Achorn, Daniel Galvez, Mark Sabini, Peter Mattson, David Kanter, Greg Diamos, Pete Warden, Josh Meyer, Vijay Janapa Reddi
Workshops
4th Robot Learning Workshop: Self-Supervised and Lifelong Learning Sponsor: Google Organizers include Alex Bewley, Vincent Vanhoucke
Differentiable Programming Workshop Sponsor: Google
Machine Learning for Creativity and Design Sponsor: Google Organizers include: Daphne Ippolito, David Ha
LatinX in AI (LXAI) Research @ NeurIPS 2021 Sponsor: Google Sponsorship Level: Platinum Workshop Chairs include: Andres Munoz Medina Mentorship Roundtables include: Jonathan Huang, Pablo Samuel Castro
Algorithmic Fairness Through the Lens of Causality and Robustness Organizers include: Jessica Schrouff, Awa Dieng
ImageNet: Past, Present, and Future Organizers include: Lucas Beyer, Xiaohua Zhai Speakers include: Emily Denton, Vittorio Ferrari, Alex Hanna, Alex Kolesnikov, Rebecca Roelofs
Optimal Transport and Machine Learning Organizers include: Marco Cuturi
Safe and Robust Control of Uncertain Systems Speakers include: Aleksandra Faust
CtrlGen: Controllable Generative Modeling in Language and Vision Speakers include: Sebastian Gehrmann
Deep Reinforcement Learning Organizers include: Chelsea Finn Speakers include: Karol Hausam, Dale Schuurmans
Distribution Shifts: Connecting Methods and Applications (DistShift) Speakers include: Chelsea Finn
ML For Systems Organizers include: Anna Goldie, Martin Maas, Azade Nazi, Azalia Mihoseini, Milad Hashemi, Kevin Swersky
Learning in Presence of Strategic Behavior Organizers include: Yishay Mansour
Bayesian Deep Learning Organizers include: Zoubin Ghahramani, Kevin Murphy
Advances in Programming Languages and Neurosymbolic Systems (AIPLANS) Organizers include: Disha Shrivastava, Vaibhav Tulsyan, Danny Tarlow
Ecological Theory of Reinforcement Learning: How Does Task Design Influence Agent Learning? Organizers include: Shixiang Shane Gu, Pablo Samuel Castro, Marc G. Bellemare
The Symbiosis of Deep Learning and Differential Equations Organizers include: Lily Hu
Out-of-Distribution Generalization and Adaptation in Natural and Artificial Intelligence Speakers include: Chelsea Finn
Cooperative AI Organizers include: Natasha Jaques
Offline Reinforcement Learning Organizers include: Rishabh Agarwal, George Tucker Speakers include: Minmin Chen
2nd Workshop on Self-Supervised Learning: Theory and Practice Organizers include: Kristina Toutanova
Data Centric AI Organizers include: Lora Aroyo
Math AI for Education (MATHAI4ED): Bridging the Gap Between Research and Smart Education Organizers include: Yuhai (Tony) Wu
Tutorials
Beyond Fairness in Machine Learning Organizers include: Emily Denton
Competitions
Evaluating Approximate Inference in Bayesian Deep Learning Organizers include: Matthew D. Hoffman, Sharad Vikram
HEAR 2021 NeurIPS Challenge Holistic Evaluation of Audio Representations Organizers include: Jesse Engel
Machine Learning for Combinatorial Optimization Organizers include: Pawel Lichocki, Miles Lubin
In recent years, pre-trained language models, such as BERT and GPT-3, have seen widespread use in natural language processing (NLP). By training on large volumes of text, language models acquire broad knowledge about the world, achieving strong performance on various NLP benchmarks. These models, however, are often opaque in that it may not be clear why they perform so well, which limits further hypothesis-driven improvement of the models. Hence, a new line of scientific inquiry has arisen: what linguistic knowledge is contained in these models?
While there are many types of linguistic knowledge that one may want to investigate, a topic that provides a strong basis for analysis is the subject–verb agreement grammar rule in English, which requires that the grammatical number of a verb agree with that of the subject. For example, the sentence “The dogs run.” is grammatical because “dogs” and “run” are both plural, but “The dogs runs.” is ungrammatical because “runs” is a singular verb.
One framework for assessing the linguistic knowledge of a language model is targeted syntactic evaluation (TSE), in which minimally different pairs of sentences, one grammatical and one ungrammatical, are shown to a model, and the model must determine which one is grammatical. TSE can be used to test knowledge of the English subject–verb agreement rule by having the model judge between two versions of the same sentence: one where a particular verb is written in its singular form, and the other in which the verb is written in its plural form.
With the above context, in “Frequency Effects on Syntactic Rule-Learning in Transformers”, published at EMNLP 2021, we investigated how a BERT model’s ability to correctly apply the English subject–verb agreement rule is affected by the number of times the words are seen by the model during pre-training. To test specific conditions, we pre-trained BERT models from scratch using carefully controlled datasets. We found that BERT achieves good performance on subject–verb pairs that do not appear together in the pre-training data, which indicates that it does learn to apply subject–verb agreement. However, the model tends to predict the incorrect form when it is much more frequent than the correct form, indicating that BERT does not treat grammatical agreement as a rule that must be followed. These results help us to better understand the strengths and limitations of pre-trained language models.
Prior Work Previous work used TSE to measure English subject–verb agreement ability in a BERT model. In this setup, BERT performs a fill-in-the-blank task (e.g., “the dog _ across the park”) by assigning probabilities to both the singular and plural forms of a given verb (e.g., “runs” and “run”). If the model has correctly learned to apply the subject–verb agreement rule, then it should consistently assign higher probabilities to the verb forms that make the sentences grammatically correct.
This previous work evaluated BERT using both natural sentences (drawn from Wikipedia) and nonce sentences, which are artificially constructed to be grammatically valid but semantically nonsensical, such as Noam Chomsky’s famous example “colorless green ideas sleep furiously”. Nonce sentences are useful when testing syntactic abilities because the model cannot just fall back on superficial corpus statistics: for example, while “dogs run” is much more common than “dogs runs”, “dogs publish” and “dogs publishes” will both be very rare, so a model is not likely to have simply memorized the fact that one of them is more likely than the other.
BERT achieves an accuracy of more than 80% on nonce sentences (far better than the random-chance baseline of 50%), which was taken as evidence that the model had learned to apply the subject–verb agreement rule. In our paper, we went beyond this previous work by pre-training BERT models under specific data conditions, allowing us to dig deeper into these results to see how certain patterns in the pre-training data affect performance.
Unseen Subject–Verb Pairs We first looked at how well the model performs on subject–verb pairs that were seen during pre-training, versus examples in which the subject and verb were never seen together in the same sentence:
BERT’s error rate increases slightly for unseen subject–verb (SV) pairs, for both natural and nonce evaluation sentences, but it is still much better than naïve heuristics, such as picking the verb form that occurred more often in the pre-training data or picking the verb form that occurred more frequently with the subject noun. This tells us that BERT is not just reflecting back the things that it sees during pre-training: making decisions based on more than just raw frequencies and generalizing to novel subject–verb pairs are indications that the model has learned to apply some underlying rule concerning subject–verb agreement.
Frequency of Verbs Next, we went beyond just seen versus unseen, and examined how the frequency of a word affects BERT’s ability to use it correctly with the subject–verb agreement rule. For this study, we chose a set of 60 verbs, and then created several versions of the pre-training data, each engineered to contain the 60 verbs at a specific frequency, ensuring that the singular and plural forms appeared the same number of times. We then trained BERT models from these different datasets and evaluated them on the subject–verb agreement task:
These results indicate that although BERT is able to model the subject–verb agreement rule, it needs to see a verb about 100 times before it can reliably use it with the rule.
Relative Frequency Between Verb Forms Finally, we wanted to understand how the relative frequencies of the singular and plural forms of a verb affect BERT’s predictions. For example, if one form of the verb (e.g., “combat”) appeared in the pre-training data much more frequently than the other verb form (e.g., “combats”), then BERT might be more likely to assign a high probability to the more frequent form, even when it is grammatically incorrect. To evaluate this, we again used the same 60 verbs, but this time we created manipulated versions of the pre-training data where the frequency ratio between verb forms varied from 1:1 to 100:1. The figure below shows BERT’s performance for these varying levels of frequency imbalance:
These results show that BERT achieves good accuracy at predicting the correct verb form when the two forms are seen the same number of times during pre-training, but the results become worse as the imbalance between the frequencies increases. This implies that even though BERT has learned how to apply subject–verb agreement, it does not necessarily use it as a “rule”, instead preferring to predict high-frequency words regardless of whether they violate the subject–verb agreement constraint.
Conclusions Using TSE to evaluate the performance of BERT reveals its linguistic abilities on syntactic tasks. Moreover, studying its syntactic ability in relation to how often words appear in the training dataset reveals the ways that BERT handles competing priorities — it knows that subjects and verbs should agree and that high frequency words are more likely, but doesn’t understand that agreement is a rule that must be followed and that the frequency is only a preference. We hope this work provides new insight into how language models reflect properties of the datasets on which they are trained.
Acknowledgements It was a privilege to collaborate with Tal Linzen and Ellie Pavlick on this project.
Most reinforcement learning (RL) and sequential decision making algorithms require an agent to generate training data through large amounts of interactions with their environment to achieve optimal performance. This is highly inefficient, especially when generating those interactions is difficult, such as collecting data with a real robot or by interacting with a human expert. This issue can be mitigated by reusing external sources of knowledge, for example, the RL Unplugged Atari dataset, which includes data of a synthetic agent playing Atari games.
However, there are very few of these datasets and a variety of tasks and ways of generating data in sequential decision making (e.g., expert data or noisy demonstrations, human or synthetic interactions, etc.), making it unrealistic and not even desirable for the whole community to work on a small number of representative datasets because these will never be representative enough. Moreover, some of these datasets are released in a form that only works with certain algorithms, which prevents researchers from reusing this data. For example, rather than including the sequence of interactions with the environment, some datasets provide a set of random interactions, making it impossible to reconstruct the temporal relation between them, while others are released in slightly different formats, which can introduce subtle bugs that are very difficult to identify.
In this context, we introduce Reinforcement Learning Datasets (RLDS), and release a suite of tools for recording, replaying, manipulating, annotating and sharing data for sequential decision making, including offline RL, learning from demonstrations, or imitation learning. RLDS makes it easy to share datasets without any loss of information (e.g., keeping the sequence of interactions instead of randomizing them) and to be agnostic to the underlying original format, enabling users to quickly test new algorithms on a wider range of tasks. Additionally, RLDS provides tools for collecting data generated by either synthetic agents (EnvLogger) or humans (RLDS Creator), as well as for inspecting and manipulating the collected data. Ultimately, integration with TensorFlow Datasets (TFDS) facilitates the sharing of RL datasets with the research community.
Dataset Structure Algorithms in RL, offline RL, or imitation learning may consume data in very different formats, and, if the format of the dataset is unclear, it's easy to introduce bugs caused by misinterpretations of the underlying data. RLDS makes the data format explicit by defining the contents and the meaning of each of the fields of the dataset, and provides tools to re-align and transform this data to fit the format required by any algorithm implementation. In order to define the data format, RLDS takes advantage of the inherently standard structure of RL datasets — i.e., sequences (episodes) of interactions (steps) between agents and environments, where agents can be, for example, rule-based/automation controllers, formal planners, humans, animals, or a combination of these. Each of these steps contains the current observation, the action applied to the current observation, the reward obtained as a result of applying action, and the discount obtained together with reward. Steps also include additional information to indicate whether the step is the first or last of the episode, or if the observation corresponds to a terminal state. Each step and episode may also contain custom metadata that can be used to store environment-related or model-related data.
Producing the Data Researchers produce datasets by recording the interactions with an environment made by any kind of agent. To maintain its usefulness, raw data is ideally stored in a lossless format by recording all the information that is produced, keeping the temporal relation between the data items (e.g., ordering of steps and episodes), and without making any assumption on how the dataset is going to be used in the future. For this, we release EnvLogger, a software library to log agent-environment interactions in an open format.
EnvLogger is an environment wrapper that records agent–environment interactions and saves them in long-term storage. Although EnvLogger is seamlessly integrated in the RLDS ecosystem, we designed it to be usable as a stand-alone library for greater modularity.
As in most machine learning settings, collecting human data for RL is a time consuming and labor intensive process. The common approach to address this is to use crowd-sourcing, which requires user-friendly access to environments that may be difficult to scale to large numbers of participants. Within the RLDS ecosystem, we release a web-based tool called RLDS Creator, which provides a universal interface to any human-controllable environment through a browser. Users can interact with the environments, e.g., play the Atari games online, and the interactions are recorded and stored such that they can be loaded back later using RLDS for analysis or to train agents.
Sharing the Data Datasets are often onerous to produce, and sharing with the wider research community not only enables reproducibility of former experiments, but also accelerates research as it makes it easier to run and validate new algorithms on a range of scenarios. For that purpose, RLDS is integrated with TensorFlow Datasets (TFDS), an existing library for sharing datasets within the machine learning community. Once a dataset is part of TFDS, it is indexed in the global TFDS catalog, making it accessible to any researcher by using tfds.load(name_of_dataset), which loads the data either in Tensorflow or in Numpy formats.
TFDS is independent of the underlying format of the original dataset, so any existing dataset with RLDS-compatible format can be used with RLDS, even if it was not originally generated with EnvLogger or RLDS Creator. Also, with TFDS, users keep ownership and full control over their data and all datasets include a citation to credit the dataset authors.
Consuming the Data Researchers can use the datasets in order to analyze, visualize or train a variety of machine learning algorithms, which, as noted above, may consume data in different formats than how it has been stored. For example, some algorithms, like R2D2 or R2D3, consume full episodes; others, like Behavioral Cloning or ValueDice, consume batches of randomized steps. To enable this, RLDS provides a library of transformations for RL scenarios. These transformations have been optimized, taking into account the nested structure of the RL datasets, and they include auto-batching to accelerate some of these operations. Using those optimized transformations, RLDS users have full flexibility to easily implement some high level functionalities, and the pipelines developed are reusable across RLDS datasets. Example transformations include statistics across the full dataset for selected step fields (or sub-fields) or flexible batching respecting episode boundaries. You can explore the existing transformations in this tutorial and see more complex real examples in this Colab.
Available Datasets At the moment, the following datasets (compatible with RLDS) are in TFDS:
Our team is committed to quickly expanding this list in the near future and external contributions of new datasets to RLDS and TFDS are welcomed.
Conclusion The RLDS ecosystem not only improves reproducibility of research in RL and sequential decision making problems, but also enables new research by making it easier to share and reuse data. We hope the capabilities offered by RLDS will initiate a trend of releasing structured RL datasets, holding all the information and covering a wider range of agents and tasks.
Acknowledgements Besides the authors of this post, this work has been done by Google Research teams in Paris and Zurich in Collaboration with Deepmind. In particular by Sertan Girgin, Damien Vincent, Hanna Yakubovich, Daniel Kenji Toyama, Anita Gergely, Piotr Stanczyk, Raphaël Marinier, Jeremiah Harmsen, Olivier Pietquin and Nikola Momchev. We also want to thank the collaboration of other engineers and researchers who provided feedback and contributed to the project. In particular, George Tucker, Sergio Gomez, Jerry Li, Caglar Gulcehre, Pierre Ruyssen, Etienne Pot, Anton Raichuk, Gabriel Dulac-Arnold, Nino Vieillard, Matthieu Geist, Alexandra Faust, Eugene Brevdo, Tom Granger, Zhitao Gong, Toby Boyd and Tom Small.
