Transcription of Improving Language Understanding by Generative Pre …
{{id}} {{{paragraph}}}
Example: marketing
Improving Language Understanding by Generative Pre …
Improving Language Understandingby Generative Pre-TrainingAlec Language Understanding comprises a wide range of diverse tasks suchas textual entailment, question answering, semantic similarity assessment, anddocument classification. Although large unlabeled text corpora are abundant,labeled data for learning these specific tasks is scarce, making it challenging fordiscriminatively trained models to perform adequately. We demonstrate that largegains on these tasks can be realized bygenerative pre-trainingof a Language modelon a diverse corpus of unlabeled text, followed bydiscriminative fine-tuningon eachspecific task. In contrast to previous approaches, we make use of task-aware inputtransformations during fine-tuning to achieve effective transfer while requiringminimal changes to the model architecture. We demonstrate the effectiveness ofour approach on a wide range of benchmarks for natural Language general task-agnostic model outperforms discriminatively trained models thatuse architectures specifically crafted for each task, significantly Improving upon thestate of the art in 9 out of the 12 tasks studied.
After training the model with the objective in Eq. 1, we adapt the parameters to the supervised target task. We assume a labeled dataset C, where each instance consists of a sequence of input tokens, x1;:::;xm, along with a label y. The inputs are passed through our pre-trained model to obtain the final transformer block’s activation hm
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: