Transcription of Generative Pretraining from Pixels - OpenAI
{{id}} {{{paragraph}}}
Generative Pretraining from PixelsMark Chen1 Alec Radford1 Rewon Child1 Jeff Wu1 Heewoo Jun1 Prafulla Dhariwal1 David Luan1 Ilya Sutskever1 AbstractInspired by progress in unsupervised representa-tion learning for natural language, we examinewhether similar models can learn useful repre-sentations for images. We train a sequence Trans-former to auto-regressively predict Pixels , withoutincorporating knowledge of the 2D input training on low-resolution ImageNet with-out labels, we find that a GPT-2 scale model learnsstrong image representations as measured by lin-ear probing, fine-tuning, and low-data classifica-tion.
Generative Pretraining from Pixels (Radford et al.,2019) formulation of the transformer de-coder block, which acts on an input tensor hlas follows: nl= layer norm(hl) al= hl+multihead attention(nl) hl+1 = al+mlp(layer norm(al)) In particular, layer norms precede both the attention and
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document:
{{id}} {{{paragraph}}}