Transcription of Training data-efficient image transformers & distillation ...
{{id}} {{{paragraph}}}
Example: marketing
Training data-efficient image transformers & distillation ...
Training data-efficient image transformers & distillation through attentionHugo Touvron?, Matthieu Cord Matthijs Douze?Francisco Massa?Alexandre Sablayrolles?Herv e J egou??Facebook AI Sorbonne UniversityAbstractRecently, neural networks purely based on attention were shown to ad-dress image understanding tasks such as image classification. These high-performing vision transformers are pre-trained with hundreds of millionsof images using a large infrastructure, thereby limiting their this work, we produce competitive convolution-free transformers bytraining on Imagenet only. We train them on a single computer in less than3 days. Our reference vision transformer (86M parameters) achieves top-1accuracy of (single-crop) on ImageNet with no external importantly, we introduce a teacher-student strategy specific totransformers.
transformers “do not generalize well when trained on insufficient amounts of data”, and the training of these models involved extensive computing resources. In this paper, we train a vision transformer on a single 8-GPU node in two to three days (53 hours of pre-training, and optionally 20 hours of fine-tuning)
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: