Blog

The latest news from Google AI

Moving Beyond Translation with the Universal Transformer

Wednesday, August 15, 2018

Posted by Stephan Gouws, Research Scientist, Google Brain Team and Mostafa Dehghani, University of Amsterdam PhD student and Google Research Internthe Transformerrecurrent neural networkssequentiallyrecurrenceabcabcabcNeural GPUNeural Turing MachineUniversal TransformersTuring completeparallel-in-time recurrence

The Universal Transformer repeatedly refines a series of vector representations (shown as h₁ to h_m) for each position of the sequence in parallel, by combining information from different positions using self-attention and applying a recurrent transition function. Arrows denote dependencies between operations.

adaptive computation mechanismI arrived at the bank after crossing the riverbankIriverbanksinglesequencedifferent transformationsbAbI linguistic reasoning taskLAMBADA language modeling taskBLEU¹hereAcknowledgementsThis research was conducted by Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Additional thanks go to Ashish Vaswani, Douglas Eck, and David Dohan for their fruitful comments and inspiration.
1 A translation quality benchmark widely used in the machine translation community, computed on the standard WMT newstest2014 English to German translation test data set.^↩