Example: barber

Learning Transferable Features with Deep Adaptation Networks

Learning Transferable Features with deep Adaptation NetworksMingsheng Long Cao Wang I. Jordan School of Software, TNList Lab for Info. Sci. & Tech., Institute for Data Science, Tsinghua University, China Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USAA bstractRecent studies reveal that a deep neural networkcan learn Transferable Features which generalizewell to novel tasks for domain Adaptation . How-ever, as deep Features eventually transition fromgeneral to specific along the network , the featuretransferability drops significantly in higher layerswith increasing domain discrepancy. Hence, it isimportant to formally reduce the dataset bias andenhance the transferability in task-specific this paper, we propose a new deep AdaptationNetwork (DAN) architecture, which generalizesdeep convolutional neural network to the domainadaptation scenario.

deep networks, resulting in statistically unboundedrisk for target tasks (Mansour et al., 2009; Ben-David et al., 2010). Our work is primarily motivated by Yosinski et al. (2014), which comprehensively explores feature transferability of deep convolutional neural networks. The method focuses on a different scenario where the learning tasks are ...

Tags:

  Network, Deep, Deep networks

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Transcription of Learning Transferable Features with Deep Adaptation Networks

1 Learning Transferable Features with deep Adaptation NetworksMingsheng Long Cao Wang I. Jordan School of Software, TNList Lab for Info. Sci. & Tech., Institute for Data Science, Tsinghua University, China Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USAA bstractRecent studies reveal that a deep neural networkcan learn Transferable Features which generalizewell to novel tasks for domain Adaptation . How-ever, as deep Features eventually transition fromgeneral to specific along the network , the featuretransferability drops significantly in higher layerswith increasing domain discrepancy. Hence, it isimportant to formally reduce the dataset bias andenhance the transferability in task-specific this paper, we propose a new deep AdaptationNetwork (DAN) architecture, which generalizesdeep convolutional neural network to the domainadaptation scenario.

2 In DAN, hidden representa-tions of all task-specific layers are embedded in areproducing kernel Hilbert space where the meanembeddings of different domain distributions canbe explicitly matched. The domain discrepancyis further reduced using an optimal multi-kernelselection method for mean embedding can learn Transferable Features with statisti-cal guarantees, and can scale linearly by unbiasedestimate of kernel embedding. Extensive empiri-cal evidence shows that the proposed architectureyields state-of-the-art image classification errorrates on standard domain Adaptation IntroductionThe generalization error of supervised Learning machineswith limited training samples will be unsatisfactorily large,while manual labeling of sufficient training data for diverseapplication domains may be prohibitive.

3 Therefore, there isincentive to establishing effective algorithms to reduce thelabeling cost, typically by leveraging off-the-shelf labeledProceedings of the32ndInternational Conference on MachineLearning, Lille, France, 2015. JMLR: W&CP volume 37. Copy-right 2015 by the author(s).data from relevant source domains to the target Adaptation addresses the problem that we have datafrom two related domains but under different domain discrepancy poses a major obstacle in adaptingpredictive models across domains. For example, an objectrecognition model trained on manually annotated imagesmay not generalize well on testing images under substantialvariations in the pose, occlusion, or illumination. Domainadaptation establishes knowledge transfer from the labeledsource domain to the unlabeled target domain by exploringdomain-invariant structures that bridge different domainsof substantial distribution discrepancy (Pan & Yang,2010).

4 One of the main approaches to establishing knowledgetransfer is to learn domain-invariant models from data,which can bridge the source and target domains in an iso-morphic latent feature space. In this direction, a fruitfullineof prior work has focused on Learning shallow Features byjointly minimizing a distance metric of domain discrepancy(Pan et al.,2011;Long et al.,2013;Baktashmotlagh et al.,2013;Gong et al.,2013;Zhang et al.,2013;Ghifary et al.,2014;Wang & Schneider,2014). However, recent studieshave shown that deep neural Networks can learn more trans-ferable Features for domain Adaptation (Glorot et al.,2011;Donahue et al.,2014;Yosinski et al.,2014), which producebreakthrough results on some domain Adaptation neural Networks are able to disentangle exploratoryfactors of variations underlying the data samples, and groupfeatures hierarchically in accordance with their relatednessto invariant factors, making representations robust to deep neural Networks are more powerful for learninggeneral and Transferable Features , the latest findings alsore-veal that the deep Features must eventually transition fromgeneral to specific along the network , and feature transfer-ability drops significantly in higher layers with increasingdomain discrepancy.

5 In other words, the Features computedin higher layers of the network must depend greatly onthe specific dataset and task (Yosinski et al.,2014), whichare task-specific Features and are not safely Transferable toLearning Transferable Features with deep Adaptation Networksnovel tasks. Another curious phenomenon is that disentan-gling the variational factors in higher layers of the networkmay enlarge the domain discrepancy, as different domainswith the new deep representations become more compact and are more mutually distinguishable (Glorot et al.,2011).Although deep Features are salient for discrimination, en-larged dataset bias may deteriorate domain Adaptation per-formance, resulting in statisticallyunboundedrisk for thetarget tasks (Mansour et al.)

6 ,2009;Ben-David et al.,2010).Inspired by the literature s latest understanding about thetransferability of deep neural Networks , we propose in thispaper a new deep Adaptation network (DAN) architecture,which generalizes deep convolutional neural network to thedomain Adaptation scenario. The main idea of this workis to enhance the feature transferability in the task-specificlayers of the deep neural network by explicitly reducingthe domain discrepancy. To establish this goal, the hiddenrepresentations of all the task-specific layers are embeddedto a reproducing kernel Hilbert space where the mean em-beddings of different domain distributions can be explicitlymatched. As mean embedding matching is sensitive to thekernel choices, an optimal multi-kernel selection procedureis devised to further reduce the domain discrepancy.

7 In ad-dition, we implement a linear-time unbiased estimate of thekernel mean embedding to enable scalable training, whichis very desirable for deep Learning . Finally, as deep modelspre-trained with large-scale repositories such as ImageNet(Russakovsky et al.,2014) are representative for general-purpose tasks (Yosinski et al.,2014;Hoffman et al.,2014),the proposed DAN model is trained by fine-tuning fromthe AlexNet model (Krizhevsky et al.,2012) pre-trained onImageNet, which is implemented in Caffe (Jia et al.,2014).Comprehensive empirical evidence demonstrates that theproposed architecture outperforms state-of-the-art resultsevaluated on the standard domain Adaptation contributions of this paper are summarized as fol-lows.

8 (1) We propose a novel deep neural network archi-tecture for domain Adaptation , in whichallthe layers cor-responding to task-specific Features are adapted in a lay-erwise manner, hence benefiting from deep Adaptation . (2) We exploremultiplekernels for adapting deep represen-tations, which substantially enhances Adaptation effective-ness compared to single kernel methods. Our model canyield unbiased deep Features with statistical Related WorkA related literature is transfer Learning (Pan & Yang,2010),which builds models that bridge different domains or tasks,explicitly taking domain discrepancy into Learning aims to mitigate the effort of manual la-beling for machine Learning (Pan et al.,2011;Gong et al.)

9 ,2013;Zhang et al.,2013;Wang & Schneider,2014) andcomputer vision (Saenko et al.,2010;Gong et al.,2012;Baktashmotlagh et al.,2013;Long et al.,2013), etc. It iswidely recognized that the domain discrepancy in the prob-ability distributions of different domains should be for-mally measured and reduced. The major bottleneck is howto match different domain distributions effectively. Mostexisting methods learn a new shallow representation modelin which the domain discrepancy can be explicitly , without Learning deep Features which can sup-press domain-specific factors, the transferability of shallowfeatures could be limited by the task-specific neural Networks learn nonlinear representations thatdisentangle and hide different explanatory factors of varia-tion behind data samples (Bengio et al.

10 ,2013). The learneddeep representations manifest invariant factors underlyingdifferent populations and are Transferable from the originaltasks to similar novel tasks (Yosinski et al.,2014). Hence, deep neural Networks have been explored for domain adap-tation (Glorot et al.,2011;Chen et al.,2012), multimodaland multi-source Learning problems (Ngiam et al.,2011;Ge et al.,2013), where significant performance gains havebeen obtained. However, all these methods depend on theassumption that deep neural Networks can learn invariantrepresentations that are Transferable across different reality, the domain discrepancy can be alleviated, butnot removed, by deep neural Networks (Glorot et al.,2011).Dataset shift has posed a bottleneck to the transferabilityofdeep Networks , resulting in statisticallyunboundedrisk fortarget tasks (Mansour et al.


Related search queries