Blog

The latest from Google Research

Transformer-XL: Unleashing the Potential of Attention Models

Tuesday, January 29, 2019

Posted by Zhilin Yang and Quoc Le, Google AIlong-range dependenceGating-based RNNsthe gradient clipping techniquenot sufficientTransformers,a fixed-length context

Vanilla Transformer with a fixed-length context at training time.

The algorithm is not able to model dependencies that are longer than a fixed length.

The segments usually do not respect the sentence boundaries, resulting in context fragmentation which leads to inefficient optimization. This is particularly troublesome even for short sequences, where long range dependency isn't an issue.

Transformer-XLSegment-level Recurrence

Transformer-XL with segment-level recurrence at training time.

Relative Positional Encodingsother relative positional encoding schemes

Vanilla Transformer with a fixed-length context at evaluation time.

Transformer-XL with segment-level recurrence at evaluation time./td>

Results

Transformer-XL learns dependency that is about 80% longer than RNNs and 450% longer than vanilla Transformers, which generally have better performance than RNNs, but are not the best for long-range dependency modeling due to fixed-length contexts (please see our paper for details).

Transformer-XL is up to 1,800+ times faster than a vanilla Transformer during evaluation on language modeling tasks, because no re-computation is needed (see figures above).

Transformer-XL has better performance in perplexity (more accurate at predicting a sample) on long sequences because of long-term dependency modeling, and also on short sequences by resolving the context fragmentation problem.