Transcription of Self-Supervised Learning
{{id}} {{{paragraph}}}
Example: marketing
Self-Supervised Learning
Self-Supervised LearningMegan LeszczynskiLecture is Self-Supervised Learning ? of self-supervision in NLP Word embeddings ( , word2vec) Language models ( , GPT) Masked language models ( , BERT) challenges Demoting bias Capturing factual knowledge Learning symbolic reasoning23 DataLabelersPretraining TaskDownstream TasksImageNet Pretrain for fine-grained image classification over 1000 classes Use feature representations for downstream tasks, detection, image segmentation, and action recognitionSupervised pretraining on large labeled, datasets has led to successful transfer Learning [Deng et al., 2009] Supervised pretraining on large labeled, datasets has led to successful transfer learning4 Across images, video, and textSNLI DatasetKinetics Dataset[Deng et al.]
•Goal: represent words as vectors for input into neural networks. •One-hot vectors? (single 1, rest 0s) pizza = [0 0 0 0 0 1 0 … 0 0 0 0 0 ] pie = [0 0 0 0 0 0 0 … 0 0 0 1 0 ] ☹Millions of words high-dimensional, sparse vectors ☹No notion of word similarity •Instead: we want a dense, low-dimensional vector for each word such that ...
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: