Deep Learning using Linear Support Vector Machines

← Back to document page

Deep Learning using Linear Support Vector MachinesYichuan of Computer Science, University of Toronto. Toronto, Ontario, , fully-connected and convolutionalneural networks have been trained to achievestate-of-the-art performance on a wide vari-ety of tasks such as speech recognition, im-age classification, natural language process-ing, and bioinformatics. For classificationtasks, most of these deep Learning modelsemploy the softmax activation function forprediction and minimize cross-entropy this paper, we demonstrate a small butconsistent advantage of replacing the soft-max layer with a Linear Support Vector ma-chine. Learning minimizes a margin-basedloss instead of the cross-entropy loss. Whilethere have been various combinations of neu-ral nets and SVMs in prior art, our resultsusing L2-SVMs show that by simply replac-ing softmax with Linear SVMs gives signifi-cant gains on popular deep Learning datasetsMNIST, CIFAR-10, and the ICML 2013 Rep-resentation Learning Workshop s face expres-sion recognition IntroductionDeep Learning using neural networks have claimedstate-of

Deep Learning using Linear Support Vector Machines Yichuan Tang tang@cs.toronto.edu Department of Computer Science, University of Toronto. Toronto, Ontario, Canada.

  Using, Linear, Machine, Learning, Support, Vector, Tang, Learning using linear support vector machines, Tang tang