Abstract - arXiv

1 [ ] 9 Mar 2015 Distilling the Knowledge in a Neural NetworkGeoffrey Hinton Google Vinyals Google DeanGoogle very simple way to improve the performance of almost any machine learningalgorithm is to train many different models on the same data and then to averagetheir predictions [3]. Unfortunately, making predictionsusing a whole ensembleof models is cumbersome and may be too computationally expensive to allow de-ployment to a large number of users, especially if the individual models are largeneural nets. Caruana and his collaborators [1] have shown that it is possible tocompress the knowledge in an ensemble into a single model which is much eas-ier to deploy and we develop this approach further using a different compressiontechnique. We achieve some surprising results on MNIST and we show that wecan significantly improve the acoustic model of a heavily used commercial systemby distilling the knowledge in an ensemble of models into a single model.

2 We alsointroduce a new type of ensemble composed of one or more full models and manyspecialist models which learn to distinguish fine-grained classes that the full mod-els confuse. Unlike a mixture of experts, these specialist models can be trainedrapidly and in IntroductionMany insects have a larval form that is optimized for extracting energy and nutrients from the envi-ronment and a completely different adult form that is optimized for the very different requirementsof traveling and reproduction. In large-scale machine learning, we typically use very similar modelsfor the training stage and the deployment stage despite their very different requirements: For taskslike speech and object recognition, training must extract structure from very large, highly redundantdatasets but it does not need to operate in real time and it canuse a huge amount of to a large number of users, however, has much morestringent requirements on latencyand computational resources.

3 The analogy with insects suggests that we should be willing to trainvery cumbersome models if that makes it easier to extract structure from the data. The cumbersomemodel could be an ensemble of separately trained models or a single very large model trained witha very strong regularizer such as dropout [9]. Once the cumbersome model has been trained, wecan then use a different kind of training, which we call distillation to transfer the knowledge fromthe cumbersome model to a small model that is more suitable for deployment. A version of thisstrategy has already been pioneered by Rich Caruana and his collaborators [1]. In their importantpaper they demonstrate convincingly that the knowledge acquired by a large ensemble of modelscan be transferred to a single small conceptual block that may have prevented more investigation of this very promising approach isthat we tend to identify the knowledge in a trained model withthe learned parameter values and thismakes it hard to see how we can change the form of the model but keep the same knowledge.

4 A moreabstract view of the knowledge, that frees it from any particular instantiation, is that it is a learned Also affiliated with the University of Toronto and the Canadian Institute for Advanced Research. Equal from input vectors to output vectors. For cumbersome models that learn to discriminatebetween a large number of classes, the normal training objective is to maximize the average logprobability of the correct answer, but a side-effect of the learning is that the trained model assignsprobabilities to all of the incorrect answers and even when these probabilities are very small, someof them are much larger than others. The relative probabilities of incorrect answers tell us a lot abouthow the cumbersome model tends to generalize. An image of a BMW, for example, may only havea very small chance of being mistaken for a garbage truck, butthat mistake is still many times moreprobable than mistaking it for a is generally accepted that the objective function used for training should reflect the true objectiveof the user as closely as possible.

5 Despite this, models are usually trained to optimize performanceon the training data when the real objective is to generalizewell to new data. It would clearlybe better to train models to generalize well, but this requires information about the correct way togeneralize and this information is not normally we are distilling the knowledgefrom a large model into a small one, however, we can train the small model to generalize in the sameway as the large model. If the cumbersome model generalizes well because, for example, it is theaverage of a large ensemble of different models, a small model trained to generalize in the same waywill typically do much better on test data than a small model that is trained in the normal way on thesame training set as was used to train the obvious way to transfer the generalization ability of thecumbersome model to a small model isto use the class probabilities produced by the cumbersome model as soft targets for training thesmall model.

6 For this transfer stage, we could use the same training set or a separate transfer the cumbersome model is a large ensemble of simpler models, we can use an arithmetic orgeometric mean of their individual predictive distributions as the soft targets. When the soft targetshave high entropy, they provide much more information per training case than hard targets and muchless variance in the gradient between training cases, so thesmall model can often be trained on muchless data than the original cumbersome model and using a muchhigher learning tasks like MNIST in which the cumbersome model almost always produces the correct answerwith very high confidence, much of the information about the learned function resides in the ratiosof very small probabilities in the soft targets. For example, one version of a 2 may be given aprobability of10 6of being a 3 and10 9of being a 7 whereas for another version it may be theother way around.

7 This is valuable information that defines arich similarity structure over the data(i. says which 2 s look like 3 s and which look like 7 s) but it has very little influence on thecross-entropy cost function during the transfer stage because the probabilities are so close to and his collaborators circumvent this problem by using the logits (the inputs to the finalsoftmax) rather than the probabilities produced by the softmax as the targets for learning the smallmodel and they minimize the squared difference between the logits produced by the cumbersomemodel and the logits produced by the small model. Our more general solution, called distillation ,is to raise the temperature of the final softmax until the cumbersome model produces a suitably softset of targets. We then use the same high temperature when training the small model to match thesesoft targets. We show later that matching the logits of the cumbersome model is actually a specialcase of transfer set that is used to train the small model could consist entirely of unlabeled data [1]or we could use the original training set.

8 We have found that using the original training set workswell, especially if we add a small term to the objective function that encourages the small modelto predict the true targets as well as matching the soft targets provided by the cumbersome , the small model cannot exactly match the soft targets and erring in the direction of thecorrect answer turns out to be DistillationNeural networks typically produce class probabilities by using a softmax output layer that convertsthe logit,zi, computed for each class into a probability,qi, by comparingziwith the other (zi/T) jexp(zj/T)(1)2whereTis a temperature that is normally set to1. Using a higher value forTproduces a softerprobability distribution over the simplest form of distillation , knowledge is transferred to the distilled model by training it ona transfer set and using a soft target distribution for each case in the transfer set that is produced byusing the cumbersome model with a high temperature in its softmax.

9 The same high temperature isused when training the distilled model, but after it has beentrained it uses a temperature of the correct labels are known for all or some of the transfer set, this method can be significantlyimproved by also training the distilled model to produce thecorrect labels. One way to do this isto use the correct labels to modify the soft targets, but we found that a better way is to simply usea weighted average of two different objective functions. The first objective function is the crossentropy with the soft targets and this cross entropy is computed using the same high temperature inthe softmax of the distilled model as was used for generatingthe soft targets from the cumbersomemodel. The second objective function is the cross entropy with the correct labels. This is computedusing exactly the same logits in softmax of the distilled model but at a temperature of 1. We foundthat the best results were generally obtained by using a condiderably lower weight on the secondobjective function.

10 Since the magnitudes of the gradients produced by the soft targets scale as1/T2it is important to multiply them byT2when using both hard and soft targets. This ensures that therelative contributions of the hard and soft targets remain roughly unchanged if the temperature usedfor distillation is changed while experimenting with Matching logits is a special case of distillationEach case in the transfer set contributes a cross-entropy gradient,dC/dzi, with respect to eachlogit,ziof the distilled model. If the cumbersome model has logitsviwhich produce soft targetprobabilitiespiand the transfer training is done at a temperature ofT, this gradient is given by: C zi=1T(qi pi) =1T(ezi/T jezj/T evi/T jevj/T)(2)If the temperature is high compared with the magnitude of thelogits, we can approximate: C zi 1T(1 +zi/TN+ jzj/T 1 +vi/TN+ jvj/T)(3)If we now assume that the logits have been zero-meaned separately for each transfer case so that jzj= jvj= 0Eq.