Deep Learning: Methods and Applications

1 Foundations and TrendsR in Signal ProcessingVol. 7, Nos. 3 4 (2013) 197 387c 2014 L. Deng and D. YuDOI: learning : Methods and ApplicationsLi DengMicrosoft ResearchOne Microsoft WayRedmond, WA 98052; YuMicrosoft ResearchOne Microsoft WayRedmond, WA 98052; Organizationofthismonograph ..2022 Some Historical Context of Deep Learning2053 Three Classes of Deep learning Athree-waycategorization .. Deep networks for unsupervised or generative learning .. Deepnetworksforsupervisedlearning .. Hybriddeepnetworks ..2264 Deep Autoencoders Unsupervised Introduction .. Use of deep autoencoders to extract speech features .. Transformingautoencoders ..2395 Pre-Trained Deep Neural Networks A Unsupervisedlayer-wisepre-training .. InterfacingDNNswithHMMs ..248iiiii6 Deep Stacking Networks and Variants Supervised Introduction.

2 A basic architecture of the deep stacking network .. AmethodforlearningtheDSNweights .. TheKernelizeddeepstackingnetwork ..2577 Selected Applications in Speech and Audio Acoustic modeling for speech recognition .. Speech synthesis .. Selected Applications in LanguageModeling and Natural Language Languagemodeling .. Selected Applications in Information Abriefintroductiontoinformationretrieval .. Use of deep stacking networks for information retrieval ..31710 Selected Applications in Object Recognitionand Computer Unsupervised or generative feature learning .. Selected Applications in Multimodaland Multi-task Multi-modalities: Speech and image .. Multi-task learning within the speech, NLP or image ..339iv12 Conclusion343 References349 AbstractThis monogrph provides an overview of general deep learning method-ology and its Applications to a variety of signal and information pro-cessing tasks.

3 The application areas are chosen with the following threecriteria in mind: (1) expertise or knowledge of the authors; (2) theapplication areas that have already been transformed by the successfuluse of deep learning technology, such as speech recognition and com-puter vision; and (3) the application areas that have the potential to beimpacted significantly by deep learning and that have been experienc-ing research growth, including natural language and text processing,information retrieval, and multimodal information processing empow-ered by multi-task deep Deng and D. learning : Methods and Applications . Foundations andTrendsR in Signal Processing, vol. 7, nos. 3 4, pp. 197 387, : Definitions and backgroundSince 2006, deep structured learning , or more commonly called deeplearning or hierarchical learning , has emerged as a new area of machinelearning research [20, 163].

4 During the past several years, the techniquesdeveloped from deep learning research have already been impactinga wide range of signal and information processing work within thetraditional and the new, widened scopes including key aspects ofmachine learning and artificial intelligence; see overview articles in[7, 20, 24, 77, 94, 161, 412], and also the media coverage of this progressin [6, 237]. A series of workshops, tutorials, and special issues or con-ference special sessions in recent years have been devoted exclusivelyto deep learning and its Applications to various signal and informationprocessing areas. These include: 2008 NIPS Deep learning Workshop; 2009 NIPS Workshop on Deep learning for Speech Recognitionand Related Applications ; 2009 ICML Workshop on learning feature Hierarchies; Definitions and background199 2011 ICML Workshop on learning Architectures, Representa-tions, and Optimization for Speech and Visual Information Pro-cessing; 2012 ICASSP Tutorial on Deep learning for Signal and Informa-tion Processing; 2012 ICML Workshop on Representation learning ; 2012 Special Section on Deep learning for Speech and LanguageProcessing in IEEE Transactions on Audio, Speech, and Lan-guage Processing (T-ASLP, January); 2010, 2011, and 2012 NIPS Workshops on Deep learning andUnsupervised feature learning ; 2013 NIPS Workshops on Deep learning and on Output Repre-sentation learning .

5 2013 Special Issue on learning Deep Architectures in IEEET ransactions on Pattern Analysis and Machine Intelligence(T-PAMI, September). 2013 International Conference on learning Representations; 2013 ICML Workshop on Representation learning Challenges; 2013 ICML Workshop on Deep learning for Audio, Speech, andLanguage Processing; 2013 ICASSP Special Session on New Types of Deep Neural Net-work learning for Speech Recognition and Related authors have been actively involved in deep learning research andin organizing or providing several of the above events, tutorials, andeditorials. In particular, they gave tutorials and invited lectures onthis topic at various places. Part of this monograph is based on theirtutorials and lecture embarking on describing details of deep learning , let s pro-vide necessary definitions.

6 Deep learning has various closely relateddefinitions or high-level descriptions: Definition 1: A class of machine learning techniques thatexploit many layers of non-linear information processing for200 Introductionsupervised or unsupervised feature extraction and transforma-tion, and for pattern analysis and classification. Definition 2: A sub-field within machine learning that is basedon algorithms for learning multiple levels of representation inorder to model complex relationships among data. Higher-levelfeatures and concepts are thus defined in terms of lower-levelones, and such a hierarchy of features is called a deep architec-ture. Most of these models are based on unsupervised learning ofrepresentations. (Wikipedia on Deep learning around March2012.) Definition 3: A sub-field of machine learning that is basedon learning several levels of representations, corresponding to ahierarchy of features or factors or concepts, where higher-levelconcepts are defined from lower-level ones, and the same lower-level concepts can help to define many higher-level concepts.

7 Deeplearning is part of a broader family of machine learning methodsbased on learning representations. An observation ( , an image)can be represented in many ways ( , a vector of pixels), butsome representations make it easier to learn tasks of interest ( ,is this the image of a human face?) from examples, and researchin this area attempts to define what makes better representationsand how to learn them. (Wikipedia on Deep learning aroundFebruary 2013.) Definition 4: Deep learning is a set of algorithms in machinelearning that attempt to learn in multiple levels, correspond-ing to different levels of abstraction. It typically uses artificialneural networks. The levels in these learned statistical modelscorrespond to distinct levels of concepts, where higher-level con-cepts are defined from lower-level ones, and the same lower-level concepts can help to define many higher-level concepts.

8 See Wikipedia on Deep learning as of this most recent update in October 2013. Definition 5: Deep learning is a new area of Machine Learningresearch, which has been introduced with the objective of movingMachine learning closer to one of its original goals: Definitions and background201 Intelligence. Deep learning is about learning multiple levels ofrepresentation and abstraction that help to make sense of datasuch as images, sound, and text. See that the deep learning that we discuss in this monograph isabout learning with deep architectures for signal and information pro-cessing. It is not about deep understanding of the signal or infor-mation, although in many cases they may be related. It should alsobe distinguished from the overloaded term in educational psychology: Deep learning describes an approach to learning that is character-ized by active engagement, intrinsic motivation, and a personal searchfor meaning.

9 Among the various high-level descriptions of deep learningabove are two key aspects: (1) models consisting of multiple layersor stages of nonlinear information processing; and (2) Methods forsupervised or unsupervised learning of feature representation atsuccessively higher, more abstract layers. Deep learning is in theintersections among the research areas of neural networks, artificialintelligence, graphical modeling, optimization, pattern recognition,and signal processing. Three important reasons for the popularityof deep learning today are the drastically increased chip processingabilities ( , general-purpose graphical processing units or GPGPUs),the significantly increased size of data used for training, and the recentadvances in machine learning and signal/information processingresearch.

10 These advances have enabled the deep learning methodsto effectively exploit complex, compositional nonlinear functions, tolearn distributed and hierarchical feature representations, and to makeeffective use of both labeled and unlabeled researchers in this area include those at University ofToronto, New York University, University of Montreal, StanfordUniversity, Microsoft Research (since 2009), Google (since about2011), IBM Research (since about 2011), Baidu (since 2012), Facebook(since 2013), UC-Berkeley, UC-Irvine, IDIAP, IDSIA, UniversityCollege London, University of Michigan, Massachusetts Institute of202 IntroductionTechnology, University of Washington, and numerous other places; more detailed list. These researchers have demonstrated empiricalsuccesses of deep learning in diverse Applications of computer vision,phonetic recognition, voice search, conversational speech recognition,speech and image feature coding, semantic utterance classifica-tion, natural language understanding, hand-writing recognition, audioprocessing, information retrieval, robotics, and even in the analysis ofmolecules that may lead to discovery of new drugs as reported recentlyby [237].