On the Sentence Embeddings from Pre-trained Language …

1 Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9119 9130,November 16 20, 2020 Association for Computational Linguistics9119On the Sentence Embeddings from Pre-trained Language ModelsBohan Li , , Hao Zhou , Junxian He , Mingxuan Wang , Yiming Yang , Lei Li ByteDance AI Lab Language Technologies Institute, Carnegie Mellon contextual representations likeBERT have achieved great success in natu-ral Language processing. However, the sen-tence Embeddings from the Pre-trained lan-guage models without fine-tuning have beenfound to poorly capture semantic meaning ofsentences. In this paper, we argue that the se-mantic information in the BERT embeddingsis not fully exploited. We first reveal the the-oretical connection between the masked lan-guage model pre-training objective and the se-mantic similarity task theoretically, and thenanalyze the BERT Sentence Embeddings em-pirically.

2 We find that BERT always inducesa non-smooth anisotropic semantic space ofsentences, which harms its performance ofsemantic address this issue,we propose to transform the anisotropic sen-tence embedding distribution to a smooth andisotropic Gaussian distribution through nor-malizing flows that are learned with an un-supervised resultsshow that our proposed BERT-flow method ob-tains significant performance gains over thestate-of-the-art Sentence Embeddings on a va-riety of semantic textual similarity tasks. Thecode is available IntroductionRecently, Pre-trained Language models and its vari-ants (Radford et al., 2019; Devlin et al., 2019; Yanget al., 2019; Liu et al., 2019) like BERT (Devlinet al., 2019) have been widely used as represen-tations of natural Language .

3 Despite their greatsuccess on many NLP tasks through fine-tuning,the Sentence Embeddings from BERT without fine-tuning are significantly inferior in terms of se-mantic textual similarity (Reimers and Gurevych, The work was done when BL was an intern ) for example, they even underperform theGloVe (Pennington et al., 2014) Embeddings whichare not contextualized and trained with a much sim-pler model. Such issues hinder applying BERT Sentence Embeddings directly to many real-worldscenarios where collecting labeled data is highly-costing or even this paper, we aim to answer two major ques-tions: (1) why do the BERT-induced Sentence em-beddings perform poorly to retrieve semanticallysimilar sentences? Do they carry too little semanticinformation, or just because the semantic meaningsin these Embeddings are not exploited properly?

4 (2)If the BERT Embeddings capture enough semanticinformation that is hard to be directly utilized, howcan we make it easier without external supervision?Towards this end, we first study the connectionbetween the BERT pretraining objective and the se-mantic similarity task. Our analysis reveals that thesentence Embeddings of BERT should be able tointuitively reflect the semantic similarity betweensentences, which contradicts with experimental ob-servations. Inspired by Gao et al. (2019) who findthat the Language modeling performance can belimited by the learned anisotropic word embeddingspace where the word Embeddings occupy a narrowcone, and Ethayarajh (2019) who find that BERT word Embeddings also suffer from anisotropy, wehypothesize that the Sentence Embeddings fromBERT as average of context Embeddings from lastlayers1 may suffer from similar issues.

5 Throughempirical probing over the Embeddings , we furtherobserve that the BERT Sentence embedding spaceis semantically non-smoothing and poorly definedin some areas, which makes it hard to be used di-rectly through simple similarity metrics such as dot1In this paper, we compute average of context embeddingsfrom last one or two layers as our Sentence Embeddings sincethey are consistently better than the [CLS] vector as shownin (Reimers and Gurevych, 2019).9120product or cosine address these issues, we propose to transformthe BERT Sentence embedding distribution into asmooth and isotropic Gaussian distribution throughnormalizing flows (Dinh et al., 2015), which isan invertible function parameterized by neural net-works. Concretely, we learn a flow-based genera-tive model to maximize the likelihood of generatingBERT Sentence Embeddings from a standard Gaus-sian latent variable in aunsupervisedfashion.

6 Dur-ing training, only the flow network is optimizedwhile the BERT parameters remain learned flow, an invertible mapping functionbetween the BERT Sentence embedding and Gaus-sian latent variable, is then used to transform theBERT Sentence embedding to the Gaussian name the proposed method perform extensive experiments on 7 stan-dard semantic textual similarity benchmarks with-out using any downstream supervision. Our empir-ical results demonstrate that the flow transforma-tion is able to consistently improve BERT by upto points with an average of points interms of spearman correlation between cosine em-bedding similarity and human annotated combined with external supervision fromnatural Language inference tasks (Bowman et al.,2015; Williams et al., 2018), our method outper-forms the Sentence -BERT Embeddings (Reimersand Gurevych, 2019), leading to new state-of-the-art performance.

7 In addition to semantic sim-ilarity tasks, we apply Sentence Embeddings toa question-answer entailment task, QNLI (Wanget al., 2019), directly without task-specific super-vision, and demonstrate the superiority of our ap-proach. Moreover, our further analysis implies thatBERT-induced similarity can excessively correlatewith lexical similarity compared to semantic sim-ilarity, and our proposed flow-based method caneffectively remedy this Understanding the SentenceEmbedding Space of BERTTo encode a Sentence into a fixed-length vector withBERT, it is a convention to either compute an aver-age of context Embeddings in the last few layers ofBERT, or extract the BERT context embedding atthe position of the [CLS] token. Note that there isno token masked when producing Sentence embed-dings, which is different from and Gurevych (2019) demonstrate thatsuch BERT Sentence Embeddings lag behind thestate-of-the-art Sentence Embeddings in terms ofsemantic similarity.

8 On the STS-B dataset, BERT Sentence Embeddings are even less competitive toaveraged GloVe (Pennington et al., 2014) embed-dings, which is a simple and non-contextualizedbaseline proposed several years ago. Nevertheless,this incompetence has not been well understoodyet in existing that as demonstrated by Reimers andGurevych (2019), averaging context embeddingsconsistently outperforms the [CLS] , unless mentioned otherwise, we use av-erage of context Embeddings as BERT sentenceembeddings and do not distinguish them in the restof the The Connection between SemanticSimilarity and BERT Pre-trainingWe consider a sequence of tokensx1:T=(x1,..,xT). Language modeling (LM) factor-izes the joint probabilityp(x1:T)in an autoregres-sive way, namelylogp(x1:T) = Tt=1logp(xt|ct)where the contextct=x1:t 1.

9 To capture bidirec-tional context during pretraining, BERT proposesa masked Language modeling (MLM) objective,which instead factorizes the probability of noisyreconstructionp( x| x) = Tt=1mtp(xt|ct), where xis a corrupted sequence, xis the masked tokens,mtis equal to 1 whenxtis masked and 0 contextct= that both LM and MLM can be reduced tomodeling the conditional distribution of a tokenxgiven the contextc, which is typically formulatedwith a softmax function as,p(x|c) =exph>cwx x exph>cwx .(1)Here the context embeddinghcis a function ofc, which is usually heavily parameterized by a deepneural network ( , a Transformer (Vaswani et al.,2017)); The word embeddingwxis a function ofx, which is parameterized by an embedding similarity between BERT Sentence embed-dings can be reduced to the similarity betweenBERT context embeddingshTchc 2.

10 However, as2 This is because we approximate BERT Sentence embed-dings with context Embeddings , and compute their dot product(or cosine similarity) as model-predicted Sentence product is equivalent to cosine similarity when the em-9121shown in Equation 1, the pretraining of BERT doesnot explicitly involve the computation ofhTchc .Therefore, we can hardly derive a mathematicalformulation of whath>chc exactly Statistics as the Proxy for Se-mantic SimilarityInstead of directly analyzinghTch c, we considerh>cwx, the dot product betweena context embeddinghcand a word to Yang et al. (2018), in a well-trainedlanguage model,h>cwxcan be approximately de-composed as follows,h>cwx logp (x|c) + c(2)=PMI(x,c) + logp(x) + c.(3)wherePMI(x,c) = logp(x,c)p(x)p(c)denotes the point-wise mutual information betweenxandc,logp(x)is a word-specific term, and cis a captures how frequently two events co-occur more than if they independently occur.