Transcription of A Gift From Knowledge Distillation: Fast Optimization ...
{{id}} {{{paragraph}}}
Example: biology
A Gift From Knowledge Distillation: Fast Optimization ...
A Gift from Knowledge Distillation: Fast Optimization , Network Minimization and Transfer LearningJunho Yim1 Donggyu Joo1 Jihoon Bae2 Junmo Kim11 School of Electrical Engineering, KAIST, South Korea2 Electronics and Telecommunications Research Institute{ , jdg105, introduce a novel technique for Knowledge transfer,where Knowledge from a pretrained deep neural network(DNN) is distilled and transferred to another DNN. As theDNN maps from the input space to the output space throughmany layers sequentially, we define the distilled knowledgeto be transferred in terms of flow between layers, which iscalculated by computing the inner product between featuresfrom two layers. When we compare the student DNN and theoriginal network with the same size as the student DNN buttrained without a teacher network, the proposed method oftransferring the distilled Knowledge as the flow between twolayers exhibits three important phenomena: (1) the studentDNN that learns the distilled Knowledge is optimized muchfaster than the original model; (2) the student DNN outper-forms the original DNN; and (3) the student DNN can learnthe distilled Knowledge from a tea}
A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning Junho Yim1 Donggyu Joo1 Jihoon Bae2 Junmo Kim1 1School of Electrical Engineering, KAIST, South Korea 2Electronics and Telecommunications Research Institute {junho.yim, jdg105, junmo.kim}@kaist.ac.kr
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: