Blog

The latest news from Google AI

Advances in Semantic Textual Similarity

Thursday, May 17, 2018

Posted by Yinfei Yang, Software Engineer and Chris Tar, Engineering Manager, Google AISmart ComposeTalk to BooksTensorFlow HubSemantic Textual SimilarityLearning Semantic Textual Similarity from Conversations

Sentences are semantically similar if they can be answered by the same responses. Otherwise, they are semantically different.

SNLIentailmentSTSBenchmarkCQA task B

For a given input, classification is considered a ranking problem against potential candidates.

Universal Sentence EncoderUniversal Sentence Encoderskip-thought

Pairwise semantic similarity comparison via outputs from TensorFlow Hub Universal Sentence Encoder.

deep average networkTransformer

Multi-task training as described in “Universal Sentence Encoder”. A variety of tasks and task structures are joined by shared encoder layers/parameters (grey boxes).

New Models modelnewTensorFlow HubUniversal Sentence Encoder - LargeUniversal Sentence Encoder - Lite

The Large model is trained with the Transformer encoder described in our second paper. It targets scenarios requiring high precision semantic representations and the best model performance at the cost of speed & size.

The Lite model is trained on a Sentence Piece vocabulary instead of words in order to significantly reduce the vocabulary size, which is a major contributor of model size. It targets scenarios where resources like memory and CPU are limited, such as on-device or browser based implementations.

AcknowledgementsDaniel Cer, Mario Guajardo-Cespedes, Sheng-Yi Kong, Noah Constant for training the models, Nan Hua, Nicole Limtiaco, Rhomni St. John for transferring tasks, Steve Yuan, Yunhsuan Sung, Brian Strope, Ray Kurzweil for discussion of the model architecture. Special thanks to Sheng-Yi Kong and Noah Constant for training the Lite model.