Transcription of Generative Pretraining from Pixels - OpenAI
{{id}} {{{paragraph}}}
Example: quiz answers
Generative Pretraining from Pixels - OpenAI
Generative Pretraining from PixelsMark Chen1 Alec Radford1 Rewon Child1 Jeff Wu1 Heewoo Jun1 Prafulla Dhariwal1 David Luan1 Ilya Sutskever1 AbstractInspired by progress in unsupervised representa-tion learning for natural language, we examinewhether similar models can learn useful repre-sentations for images. We train a sequence Trans-former to auto-regressively predict Pixels , withoutincorporating knowledge of the 2D input training on low-resolution ImageNet with-out labels, we find that a GPT-2 scale model learnsstrong image representations as measured by lin-ear probing, fine-tuning, and low-data classifica-tion. On CIFAR-10, we achieve accuracywith a linear probe, outperforming a supervisedWide ResNet, and accuracy with full fine-tuning, matching the top supervised pre-trainedmodels. An even larger model trained on a mix-ture of ImageNet and web images is competitivewith self-supervised benchmarks on ImageNet,achieving top-1 accuracy on a linear probeof our IntroductionUnsupervised pre-training played a central role in the resur-gence of deep learning.
First, we pre-process raw images by resizing to a low resolution and reshaping into a 1D sequence. We then chose one of two pre-training objectives, auto-regressive next pixel prediction or masked pixel prediction. Finally, we evaluate ... We learn a projection from fLto class logits, which we use to minimize a cross entropy loss L CLF.
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: