Transcription of Self-Supervised Learning - Stanford University
{{id}} {{{paragraph}}}
Example: biology
Self-Supervised Learning - Stanford University
Self-Supervised LearningMegan LeszczynskiLecture is Self-Supervised Learning ? of self-supervision in NLP Word embeddings ( , word2vec) language models ( , GPT) Masked language models ( , BERT) challenges Demoting bias Capturing factual knowledge Learning symbolic reasoning23 DataLabelersPretraining TaskDownstream TasksImageNet Pretrain for fine-grained image classification over 1000 classes Use feature representations for downstream tasks, detection, image segmentation, and action recognitionSupervised pretraining on large labeled, datasets has led to successful transfer Learning [Deng et al., 2009] Supervised pretraining on large labeled, datasets has led to successful transfer learning4 Across images, video, and textSNLI DatasetKinetics Dataset[Deng et al.]
•Language models (e.g., GPT) •Masked language models (e.g., BERT) 3. Open challenges •Demoting bias •Capturing factual knowledge •Learning symbolic reasoning 2. 3 Data Labelers Pretraining Task Downstream Tasks ... •Loss function (skip-gram): For a corpus with !words, ...
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: