A Gift From Knowledge Distillation: Fast Optimization ...
A Gift from Knowledge Distillation: Fast Optimization , Network Minimization and transfer LearningJunho Yim1Donggyu Joo1Jihoon Bae2Junmo Kim11School of Electrical Engineering, KAIST, South Korea2Electronics and Telecommunications Research Institute{ , jdg105, introduce a novel technique for Knowledge transfer ,where Knowledge from a pretrained deep neural network(DNN) is distilled and transferred to another DNN. As theDNN maps from the input space to the output space throughmany layers sequentially, we define the distilled knowledgeto be transferred in terms of flow between layers, which iscalculated by computing the inner product between featuresfrom two layers.}
work and a network using various knowledge transfer tech-niques. The third task was transfer learning. Although a new task may provide only a small dataset, transfer learning can take advantage of a deep and heavy DNN pretrained with a huge dataset [2]. Because our proposed method has the advantage of being able to transfer the distilled knowledge
Download A Gift From Knowledge Distillation: Fast Optimization ...
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: