BART: Denoising Sequence-to-Sequence Pre-training for ...

1 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871 7880 July 5 - 10, 2020 Association for Computational Linguistics7871 BART: Denoising Sequence-to-Sequence Pre-training for NaturalLanguage Generation, Translation, and ComprehensionMike Lewis*, Yinhan Liu*, Naman Goyal*, Marjan Ghazvininejad,Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke ZettlemoyerFacebook present BART, a Denoising autoencoderfor pretraining Sequence-to-Sequence is trained by (1) corrupting text with anarbitrary noising function, and (2) learning amodel to reconstruct the original text. It usesa standard Tranformer-based neural machinetranslation architecture which, despite its sim-plicity, can be seen as generalizing BERT (dueto the bidirectional encoder), GPT (with theleft-to-right decoder), and other recent Pre-training schemes.

2 We evaluate a number ofnoising approaches, finding the best perfor-mance by both randomly shuffling the order ofsentences and using a novel in-filling scheme,where spans of text are replaced with a sin-gle mask is particularly ef-fective when fine tuned for text generationbut also works well for comprehension matches the performance of RoBERTa onGLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive di-alogue, question answering, and summariza-tion tasks, with gains of up to also provides a BLEU increase overa back-translation system for machine transla-tion, with only target language pretraining. Wealso replicate other pretraining schemes withinthe BART framework, to understand their ef-fect on end-task IntroductionSelf-supervised methods have achieved remarkablesuccess in a wide range of NLP tasks (Mikolov et al.)

3 ,2013; Peters et al., 2018; Devlin et al., 2019; Joshiet al., 2019; Yang et al., 2019; Liu et al., 2019).The most successful approaches have been variants ofmasked language models, which are Denoising autoen-coders that are trained to reconstruct text where a ran-dom subset of the words has been masked out. Recentwork has shown gains by improving the distribution of1 Code and pre-trained models for BART are avail-able tokens (Joshi et al., 2019), the order in whichmasked tokens are predicted (Yang et al., 2019), and theavailable context for replacing masked tokens (Donget al., 2019). However, these methods typically focuson particular types of end tasks ( span prediction,generation, etc.

4 , limiting their this paper, we present BART, which pre-trainsa model combining Bidirectional and Auto-RegressiveTransformers. BART is a Denoising autoencoder builtwith a Sequence-to-Sequence model that is applicableto a very wide range of end tasks. Pretraining hastwo stages (1) text is corrupted with an arbitrary nois-ing function, and (2) a Sequence-to-Sequence model islearned to reconstruct the original text. BART uses astandard Tranformer-based neural machine translationarchitecture which, despite its simplicity, can be seen asgeneralizing BERT (due to the bidirectional encoder),GPT (with the left-to-right decoder), and many othermore recent pretraining schemes (see Figure 1).A key advantage of this setup is the noising flexibil-ity; arbitrary transformations can be applied to the orig-inal text, including changing its length.

5 We evaluatea number of noising approaches, finding the best per-formance by both randomly shuffling the order of theoriginal sentences and using a novel in-filling scheme,where arbitrary length spans of text (including zerolength) are replaced with a single mask token. This ap-proach generalizes the original word masking and nextsentence prediction objectives in BERT by forcing themodel to reason more about overall sentence length andmake longer range transformations to the is particularly effective when fine tuned fortext generation but also works well for comprehen-sion tasks. It matches the performance of RoBERTa(Liu et al., 2019) with comparable training resourceson GLUE (Wang et al.)

6 , 2018) and SQuAD (Rajpurkaret al., 2016), and achieves new state-of-the-art resultson a range of abstractive dialogue, question answering,and summarization tasks. For example, it improvesperformance by ROUGE over previous work onXSum (Narayan et al., 2018).BART also opens up new ways of thinking about finetuning. We present a new scheme for machine transla-tion where a BART model is stacked above a few ad-ditional transformer layers. These layers are trained7872 Bidirectional EncoderA _ C _ E B D (a) BERT: Random tokens are replaced with masks, andthe document is encoded bidirectionally. Missing tokensare predicted independently, so BERT cannot easily beused for DecoderA B C D E<s> A B C D (b) GPT: Tokens are predicted auto-regressively, meaningGPT can be used for generation.

7 However words can onlycondition on leftward context, so it cannot learn bidirec-tional DecoderBidirectional EncoderA B C D EA _ B _ E <s> A B C D (c) BART: Inputs to the encoder need not be aligned with decoder outputs, allowing arbitary noise transformations. Here, adocument has been corrupted by replacing spans of text with a mask symbols. The corrupted document (left) is encoded witha bidirectional model, and then the likelihood of the original document (right) is calculated with an autoregressive fine-tuning, an uncorrupted document is input to both the encoder and decoder, and we use representations from the finalhidden state of the 1: A schematic comparison of BART with BERT (Devlin et al.)

8 , 2019) and GPT (Radford et al., 2018).to essentially translate the foreign language to noisedEnglish, by propagation through BART, thereby us-ing BART as a pre-trained target-side language approach improves performance over a strongback-translation MT baseline by BLEU on theWMT Romanian-English better understand these effects, we also reportan ablation analysis that replicates other recently pro-posed training objectives. This study allows us to care-fully control for a number of factors, including dataand optimization parameters, which have been shownto be as important for overall performance as the se-lection of training objectives (Liu et al., 2019). We findthat BART exhibits the most consistently strong perfor-mance across the full range of tasks we ModelBART is a Denoising autoencoder that maps a corrupteddocument to the original document it was derived is implemented as a Sequence-to-Sequence modelwith a bidirectional encoder over corrupted text and aleft-to-right autoregressive decoder.

9 For Pre-training ,we optimize the negative log likelihood of the ArchitectureBART uses the standard Sequence-to-Sequence Trans-former architecture from (Vaswani et al., 2017), ex-cept, following GPT, that we modify ReLU activa-tion functions to GeLUs (Hendrycks & Gimpel, 2016)and initialise parameters fromN(0, ). For ourbase model, we use 6 layers in the encoder and de-coder, and for our large model we use 12 layers ineach. The architecture is closely related to that used inBERT, with the following differences: (1) each layer ofthe decoder additionally performs cross-attention overthe final hidden layer of the encoder (as in the trans-former Sequence-to-Sequence model); and (2) BERT uses an additional feed-forward network before word-prediction, which BART does not.

10 In total, BART con-tains roughly 10% more parameters than the equiva-lently sized BERT Pre-training BARTBART is trained by corrupting documents and then op-timizing a reconstruction loss the cross-entropy be-tween the decoder s output and the original existing Denoising autoencoders, which are tai-lored to specific noising schemes, BART allows us toapplyanytype of document corruption. In the extremecase, where all information about the source is lost,BART is equivalent to a language experiment with several previously proposed andnovel transformations, but we believe there is a sig-nificant potential for development of other new alter-natives. The transformations we used are summarizedbelow, and examples are shown in Figure MaskingFollowing BERT (Devlin et al.)