BERT: Pre-training of Deep Bidirectional Transformers for ...

Proceedings of NAACL-HLT 2019, pages 4171 4186 Minneapolis, Minnesota, June 2 - June 7, 2019 Association for Computational Linguistics4171 BERT: Pre-training of Deep Bidirectional Transformers forLanguage UnderstandingJacob Devlin Ming-Wei Chang Kenton Lee Kristina ToutanovaGoogle AI introduce a new language representa-tion model calledBERT, which stands forBidirectionalEncoderRepresentations fromTransformers. Unlike recent language repre-sentation models (Peters et al., 2018a; Rad-ford et al., 2018), BERT is designed to pre-train deep Bidirectional representations fromunlabeled text by jointly conditioning on bothleft and right context in all layers. As a re-sult, the pre-trained BERT model can be fine-tuned with just one additional output layerto create state-of-the-art models for a widerange of tasks, such as question answering andlanguage inference, without substantial task-specific architecture is conceptually simple and empiricallypowerful.

tion answering (Rajpurkar et al.,2016), sentiment analysis (Socher et al.,2013), and named entity recognition (Tjong Kim Sang and De Meulder, 2003).Melamud et al.(2016) proposed learning contextual representations through a task to pre-dict a single word from both left and right context using LSTMs. Similar to ELMo, their model is

Tags:

  Analysis, Learning, Words, Sentiment, Sentiment analysis

Information