Sentence-BERT: Sentence Embeddings using Siamese BERT …

1 Sentence -BERT: Sentence Embeddings using Siamese BERT-NetworksNils Reimers and Iryna GurevychUbiquitous Knowledge Processing Lab (UKP-TUDA)Department of Computer Science, Technische Universit at (Devlin et al., 2018) and RoBERTa (Liuet al., 2019) has set a new state-of-the-artperformance on Sentence -pair regression taskslike semantic textual similarity (STS). How-ever, it requires that both sentences are fedinto the network, which causes a massive com-putational overhead: Finding the most sim-ilar pair in a collection of 10,000 sentencesrequires about 50 million inference computa-tions (~65 hours) with BERT. The constructionof BERT makes it unsuitable for semantic sim-ilarity search as well as for unsupervised taskslike this publication, we present Sentence -BERT(SBERT), a modification of the pretrainedBERT network that use Siamese and triplet net-work structures to derive semantically mean-ingful Sentence Embeddings that can be com-pared using cosine-similarity.

2 This reduces theeffort for finding the most similar pair from 65hours with BERT / RoBERTa to about 5 sec-onds with SBERT, while maintaining the ac-curacy from evaluate SBERT and SRoBERTa on com-mon STS tasks and transfer learning tasks,where it outperforms other state-of-the-artsentence Embeddings IntroductionIn this publication, we present Sentence -BERT(SBERT), a modification of the BERT network us-ing Siamese and triplet networks that is able toderive semantically meaningful Sentence embed-dings2. This enables BERT to be used for certainnew tasks, which up-to-now were not applicablefor BERT. These tasks include large-scale seman-1 Code available: meaningfulwe mean that semanticallysimilar sentences are close in vector similarity comparison, clustering, and informa-tion retrieval via semantic set new state-of-the-art performance onvarious Sentence classification and Sentence -pairregression tasks.

3 BERT uses a cross-encoder: Twosentences are passed to the transformer networkand the target value is predicted. However, thissetup is unsuitable for various pair regression tasksdue to too many possible combinations. Findingin a collection ofn= 10 000sentences the pairwith the highest similarity requires with BERTn (n 1)/2 = 49 995 000inference a modern V100 GPU, this requires about 65hours. Similar, finding which of the over 40 mil-lion existent questions of Quora is the most similarfor a new question could be modeled as a pair-wisecomparison with BERT, however, answering a sin-gle query would require over 50 common method to address clustering and se-mantic search is to map each Sentence to a vec-tor space such that semantically similar sentencesare close. Researchers have started to input indi-vidual sentences into BERT and to derive fixed-size Sentence Embeddings .

4 The most commonlyused approach is to average the BERT output layer(known as BERT Embeddings ) or by using the out-put of the first token (the[CLS]token). As wewill show, this common practice yields rather badsentence Embeddings , often worse than averagingGloVe Embeddings (Pennington et al., 2014).To alleviate this issue, we developed Siamese network architecture enables thatfixed-sized vectors for input sentences can be de-rived. using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, se-mantically similar sentences can be found. Thesesimilarity measures can be performed extremelyefficient on modern hardware, allowing SBERTto be used for semantic similarity search as wellas for clustering. The complexity for finding [ ] 27 Aug 2019most similar Sentence pair in a collection of 10,000sentences is reduced from 65 hours with BERT tothe computation of 10,000 Sentence Embeddings (~5 seconds with SBERT) and computing cosine-similarity (~ seconds).

5 By using optimizedindex structures, finding the most similar Quoraquestion can be reduced from 50 hours to a fewmilliseconds (Johnson et al., 2017).We fine-tune SBERT on NLI data, which cre-ates Sentence Embeddings that significantly out-perform other state-of-the-art Sentence embeddingmethods like InferSent (Conneau et al., 2017) andUniversal Sentence Encoder (Cer et al., 2018). Onseven Semantic Textual Similarity (STS) tasks,SBERT achieves an improvement of pointscompared to InferSent and points compared toUniversal Sentence Encoder. On SentEval (Con-neau and Kiela, 2018), an evaluation toolkit forsentence Embeddings , we achieve an improvementof and points, can be adapted to a specific task. Itsets new state-of-the-art performance on a chal-lenging argument similarity dataset (Misra et al.)

6 ,2016) and on a triplet dataset to distinguish sen-tences from different sections of a Wikipedia arti-cle (Dor et al., 2018).The paper is structured in the following way:Section 3 presents SBERT, section 4 evaluatesSBERT on common STS tasks and on the chal-lenging Argument Facet Similarity (AFS) corpus(Misra et al., 2016). Section 5 evaluates SBERTon SentEval. In section 6, we perform an ablationstudy to test some design aspect of SBERT. In sec-tion 7, we compare the computational efficiency ofSBERT Sentence Embeddings in contrast to otherstate-of-the-art Sentence embedding Related WorkWe first introduce BERT, then, we discuss state-of-the-art Sentence embedding (Devlin et al., 2018) is a pre-trainedtransformer network (Vaswani et al., 2017), whichset for various NLP tasks new state-of-the-art re-sults, including question answering, Sentence clas-sification, and Sentence -pair regression.

7 The inputfor BERT for Sentence -pair regression consists ofthe two sentences, separated by a special[SEP]token. Multi-head attention over 12 (base-model)or 24 layers (large-model) is applied and the out-put is passed to a simple regression function to de-rive the final label. using this setup, BERT set anew state-of-the-art performance on the SemanticTextual Semilarity (STS) benchmark (Cer et al.,2017). RoBERTa (Liu et al., 2019) showed, thatthe performance of BERT can further improved bysmall adaptations to the pre-training process. Wealso tested XLNet (Yang et al., 2019), but it led ingeneral to worse results than large disadvantage of the BERT networkstructure is that no independent Sentence embed-dings are computed, which makes it difficult to de-rive Sentence Embeddings from BERT.

8 To bypassthis limitations, researchers passed single sen-tences through BERT and then derive a fixed sizedvector by either averaging the outputs (similar toaverage word Embeddings ) or by using the outputof the specialCLStoken (for example: May et al.(2019); Zhang et al. (2019); Qiao et al. (2019)).These two options are also provided by the popu-lar bert-as-a-service-repository3. Up to our knowl-edge, there is so far no evaluation if these methodslead to useful Sentence Embeddings are a well studied areawith dozens of proposed methods. Skip-Thought(Kiros et al., 2015) trains an encoder-decoder ar-chitecture to predict the surrounding (Conneau et al., 2017) uses labeleddata of the Stanford Natural Language Inferencedataset (Bowman et al., 2015) and the Multi-Genre NLI dataset (Williams et al.)

9 , 2018) to traina Siamese BiLSTM network with max-poolingover the et al. showed, thatInferSent consistently outperforms unsupervisedmethods like SkipThought. Universal SentenceEncoder (Cer et al., 2018) trains a transformernetwork and augments unsupervised learning withtraining on SNLI. Hill et al. (2016) showed, thatthe task on which Sentence Embeddings are trainedsignificantly impacts their quality. Previous work(Conneau et al., 2017; Cer et al., 2018) found thatthe SNLI datasets are suitable for training sen-tence Embeddings . Yang et al. (2018) presenteda method to train on conversations from Redditusing Siamese DAN and Siamese transformer net-works, which yielded good results on the STSbenchmark et al. (2019) addresses the run-timeoverhead of the cross-encoder from BERT andpresent a method (poly-encoders) to computea score betweenmcontext vectors and pre-3 A Sentence B BERT BERT u v pooling pooling (u, v, |u-v|) Softmax classifier Figure 1: SBERT architecture with classification ob-jective function, , for fine-tuning on SNLI two BERT networks have tied weights (siamesenetwork structure).

10 Computed candidate Embeddings using idea works for finding the highest scoringsentence in a larger collection. However, poly-encoders have the drawback that the score functionis not symmetric and the computational overheadis too large for use-cases like clustering, whichwould requireO(n2)score neural Sentence embedding methodsstarted the training from a random this publication, we use the pre-trained BERTand RoBERTa network and only fine-tune it toyield useful Sentence Embeddings . This reducessignificantly the needed training time: SBERT canbe tuned in less than 20 minutes, while yieldingbetter results than comparable Sentence embed-ding ModelSBERT adds a pooling operation to the outputof BERT / RoBERTa to derive a fixed sized sen-tence embedding.