Transcription of BART: Denoising Sequence-to-Sequence Pre-training for ...
{{id}} {{{paragraph}}}
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871 7880 July 5 - 10, 2020 Association for Computational Linguistics7871 BART: Denoising Sequence-to-Sequence Pre-training for NaturalLanguage Generation, Translation, and ComprehensionMike Lewis*, Yinhan Liu*, Naman Goyal*, Marjan Ghazvininejad,Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke ZettlemoyerFacebook present BART, a Denoising autoencoderfor pretraining Sequence-to-Sequence is trained by (1) corrupting text with anarbitrary noising function, and (2) learning amodel to reconstruct the original text. It usesa standard Tranformer-based neural machinetranslation architecture which, despite its sim-plicity, can be seen as generalizing BERT (dueto the bidirectional encoder), GPT (with theleft-to-right decoder), and other recent Pre-training schemes.
BART is a denoising autoencoder that maps a corrupted document to the original document it was derived from. It is implemented as a sequence-to-sequence model with a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder. For pre-training,
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document:
{{id}} {{{paragraph}}}