Blog

The latest from Google Research

More Efficient NLP Model Pre-training with ELECTRA

Tuesday, March 10, 2020

Posted by Kevin Clark, Student Researcher and Thang Luong, Senior Research Scientist, Google Research, Brain TeamBERTRoBERTaXLNetALBERTT5sentiment analysisquestion answeringGPT

Existing pre-training methods and their disadvantages. Arrows indicate which tokens are used to produce a given output representation (rectangle). Left: Traditional language models (e.g., GPT) only use context to the left of the current word. Right: Masked language models (e.g., BERT) use context from both the left and right, but predict only a small subset of words for each input.

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsELECTRAGLUESQuADreleasedMaking Pre-training Fasterreplaced token detectiongenerative adversarial networksdiscriminatorevery

Replaced token detection trains a bidirectional model while learning from all input positions.

generatordifficultytransformer

Further details on the replaced token detection (RTD) task. The fake tokens are sampled from a small masked language model that is trained jointly with ELECTRA.

ELECTRA Results

The x-axis shows the amount of compute used to train the model (measured in FLOPs) and the y-axis shows the dev GLUE score. ELECTRA learns much more efficiently than existing pre-trained NLP models. Note that current best models on GLUE such as T5 (11B) do not fit on this plot because they use much more compute than others (around 10x more than RoBERTa).