Blog

The latest from Google Research

Advancing Semi-supervised Learning with Unsupervised Data Augmentation

Wednesday, July 10, 2019

Posted by Qizhe Xie, Student Researcher and Thang Luong, Senior Research Scientist, Google Research, Brain TeamGPUTPUImageNetnatural language processingvisionspeechsupervised learning

Example augmentation operations for text-based (top) or image-based (bottom) training data.

Unsupervised Data Augmentation (UDA) for Consistency Trainingsemi-supervised learningrevival of semi-supervised learningBERTgithubUnsupervised Data Augmentation Explainedloss function

An overview of Unsupervised Data Augmentation (UDA). Left: Standard supervised loss is computed when labeled data is available. Right: With unlabeled data, a consistency loss is computed between an example and its augmented version.

Gaussian noiseadversarial noiseback translationAutoAugmentTF-IDFBenchmarks in NLP and Computer VisionIMDb

Benchmark on IMDb, a sentiment analysis task. UDA surpasses state-of-the-art results in supervised learning across different training sizes.

CIFAR-10VATICTPyramidNet+ShakeDropSVHN

SSL benchmark on CIFAR-10 and SVHN image classification tasks. UDA surpases existing semi-supervised learning methods.

ImageNetReleasecodebaseAcknowledgementsSpecial thanks to the co-authors of the paper Zihang Dai, Eduard Hovy, and Quoc V. Le. We’d also like to thank Hieu Pham, Adams Wei Yu, Zhilin Yang, Colin Raffel, Olga Wichrowska, Ekin Dogus Cubuk, Guokun Lai, Jiateng Xie, Yulun Du, Trieu Trinh, Ran Zhao, Ola Spyra, Brandon Yang, Daiyi Peng, Andrew Dai, Samy Bengio and Jeff Dean for their help with this project. A preprint is available online.

In addition, we would also like to refer readers to other approaches on semi-supervised learning concurrently developed by our colleagues in Google Research, such as MixMatch and S4L, that have demonstrated promising results. We thank the authors of the MixMatch and S4L papers for their comments and feedback on our paper and blog post.