Blog

The latest from Google Research

Encode, Tag and Realize: A Controllable and Efficient Approach for Text Generation

Friday, January 31, 2020

Posted by Eric Malmi and Sebastian Krause, Software Engineers, Google ResearchSequence-to-sequencesummarizationsentence fusionTransformerunsupervised pre-traininghallucinationEncode, Tag, Realize: High-Precision Text Editingopen sourcedrealizationDesign and Functionality of LaserTaggerKeepDeleteKeep-AddXDelete-AddXX

LaserTagger applied to sentence fusion. The predicted edit operations correspond to deleting “. Turing" and adding "and he" before it. Notice the high overlap between the input and output text.

ResultsBERTWikiSplit

When training the models on the full dataset of 1 million examples, both LaserTagger and a BERT-based seq2seq baseline model perform comparably, but when training on a subsample of 10,000 examples or less, LaserTagger clearly outperforms the baseline model (the higher the SARI score the better).

Key Advantages of LaserTagger

Control: By controlling the output phrase vocabulary, which we can also manually edit or curate, LaserTagger is less susceptible to hallucination than the seq2seq baseline.

Inference speed: LaserTagger computes predictions up to 100 times faster than the seq2seq baseline, making it suitable for real-time applications.

Data efficiency: LaserTagger produces reasonable outputs, even when trained using only a few hundred or a few thousand training examples. In our experiments, a competitive seq2seq baseline required tens of thousands of examples to obtain comparable performance.

Why This Mattersthey grow longer, become more complex, and come as part of a dialogue discourseGitHub repoAcknowledgementsThis research was conducted by Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. We are grateful for useful discussions with Enrique Alfonseca, Idan Szpektor, and Orgad Keller.