Deep Learning

1 The essence of knowledgeFnT SIG 7:3-4 Deep Learning ; Methods and Applications Li Deng and Dong YuDeep LearningMethods and ApplicationsLi Deng and Dong YuDeep Learning : Methods and Applications provides an overview of general deep Learning methodology and its applications to a variety of signal and information processing tasks. The application areas are chosen with the following three criteria in mind: (1) expertise or knowledge of the authors; (2) the application areas that have already been transformed by the successful use of deep Learning technology, such as speech recognition and computer vision; and (3) the application areas that have the potential to be impacted significantly by deep Learning and that have been benefitting from recent research efforts, including natural language and text processing, information retrieval, and multimodal information processing empowered by multi-task deep Learning : Methods and Applications is a timely and important book for researchers and students with an interest in deep Learning methodology and its applications in signal and information processing.

2 This book provides an overview of a sweeping range of up-to-date deep Learning methodologies and their application to a variety of signal and information processing tasks, including not only automatic speech recognition (ASR), but also computer vision, language modeling, text processing, multimodal Learning , and information retrieval. This is the first and the most valuable book for deep and wide Learning of deep Learning , not to be missed by anyone who wants to know the breathtaking impact of deep Learning on many facets of information processing, especially ASR, all of vital importance to our modern technological society. Sadaoki Furui, President of Toyota Technological Institute at Chicago, and Professor at the Tokyo Institute of TechnologyFoundations and Trends inSignal Processing7:3-4 Deep LearningMethods and ApplicationsLi Deng and Dong YunownowThis book is originally published asFoundations and Trends in Signal ProcessingVolume 7 Issues 3-4, ISSN: essence of knowledgeFnT SIG 7:3-4 Deep Learning ; Methods and Applications Li Deng and Dong YuDeep LearningMethods and ApplicationsLi Deng and Dong YuDeep Learning : Methods and Applications provides an overview of general deep Learning methodology and its applications to a variety of signal and information processing tasks.

3 The application areas are chosen with the following three criteria in mind: (1) expertise or knowledge of the authors; (2) the application areas that have already been transformed by the successful use of deep Learning technology, such as speech recognition and computer vision; and (3) the application areas that have the potential to be impacted significantly by deep Learning and that have been benefitting from recent research efforts, including natural language and text processing, information retrieval, and multimodal information processing empowered by multi-task deep Learning : Methods and Applications is a timely and important book for researchers and students with an interest in deep Learning methodology and its applications in signal and information processing. This book provides an overview of a sweeping range of up-to-date deep Learning methodologies and their application to a variety of signal and information processing tasks, including not only automatic speech recognition (ASR), but also computer vision, language modeling, text processing, multimodal Learning , and information retrieval.

4 This is the first and the most valuable book for deep and wide Learning of deep Learning , not to be missed by anyone who wants to know the breathtaking impact of deep Learning on many facets of information processing, especially ASR, all of vital importance to our modern technological society. Sadaoki Furui, President of Toyota Technological Institute at Chicago, and Professor at the Tokyo Institute of TechnologyFoundations and Trends inSignal Processing7:3-4 Deep LearningMethods and ApplicationsLi Deng and Dong YunownowThis book is originally published asFoundations and Trends in Signal ProcessingVolume 7 Issues 3-4, ISSN: and TrendsR in Signal ProcessingVol. 7, Nos. 3 4 (2013) 197 387c 2014 L. Deng and D. YuDOI: Learning : Methods and ApplicationsLi DengMicrosoft ResearchOne Microsoft WayRedmond, WA 98052; YuMicrosoft ResearchOne Microsoft WayRedmond, WA 98052; Organizationofthismonograph.

5 2022 Some Historical Context of Deep Learning2053 Three Classes of Deep Learning Athree-waycategorization .. Deep networks for unsupervised or generative Learning .. Deepnetworksforsupervisedlearning .. Hybriddeepnetworks ..2264 Deep Autoencoders Unsupervised Introduction .. Use of deep autoencoders to extract speech features .. Transformingautoencoders ..2395 Pre-Trained Deep Neural Networks A Unsupervisedlayer-wisepre-training .. InterfacingDNNswithHMMs ..248iiiii6 Deep Stacking Networks and Variants Supervised Introduction .. A basic architecture of the deep stacking network .. AmethodforlearningtheDSNweights .. TheKernelizeddeepstackingnetwork ..2577 Selected Applications in Speech and Audio Acoustic modeling for speech recognition .. Speech synthesis .. Selected Applications in LanguageModeling and Natural Language Languagemodeling .. Selected Applications in Information Abriefintroductiontoinformationretrieval .

6 Use of deep stacking networks for information retrieval ..31710 Selected Applications in Object Recognitionand Computer Unsupervised or generative feature Learning .. Selected Applications in Multimodaland Multi-task Multi-modalities: Speech and image .. Multi-task Learning within the speech, NLP or image ..339iv12 Conclusion343 References349 AbstractThis monograph provides an overview of general deep Learning method-ology and its applications to a variety of signal and information pro-cessing tasks. The application areas are chosen with the following threecriteria in mind: (1) expertise or knowledge of the authors; (2) theapplication areas that have already been transformed by the successfuluse of deep Learning technology, such as speech recognition and com-puter vision; and (3) the application areas that have the potential to beimpacted significantly by deep Learning and that have been experienc-ing research growth, including natural language and text processing,information retrieval, and multimodal information processing empow-ered by multi-task deep Deng and D.

7 Learning : Methods and Applications. Foundations andTrendsR in Signal Processing, vol. 7, nos. 3 4, pp. 197 387, : Definitions and backgroundSince 2006, deep structured Learning , or more commonly called deeplearning or hierarchical Learning , has emerged as a new area of machinelearning research [20, 163]. During the past several years, the techniquesdeveloped from deep Learning research have already been impactinga wide range of signal and information processing work within thetraditional and the new, widened scopes including key aspects ofmachine Learning and artificial intelligence; see overview articles in[7, 20, 24, 77, 94, 161, 412], and also the media coverage of this progressin [6, 237]. A series of workshops, tutorials, and special issues or con-ference special sessions in recent years have been devoted exclusivelyto deep Learning and its applications to various signal and informationprocessing areas.

8 These include: 2008 NIPS Deep Learning Workshop; 2009 NIPS Workshop on Deep Learning for Speech Recognitionand Related Applications; 2009 ICML Workshop on Learning Feature Hierarchies; Definitions and background199 2011 ICML Workshop on Learning Architectures, Representa-tions, and Optimization for Speech and Visual Information Pro-cessing; 2012 ICASSP Tutorial on Deep Learning for Signal and Informa-tion Processing; 2012 ICML Workshop on Representation Learning ; 2012 Special Section on Deep Learning for Speech and LanguageProcessing in IEEE Transactions on Audio, Speech, and Lan-guage Processing (T-ASLP, January); 2010, 2011, and 2012 NIPS Workshops on Deep Learning andUnsupervised Feature Learning ; 2013 NIPS Workshops on Deep Learning and on Output Repre-sentation Learning ; 2013 Special Issue on Learning Deep Architectures in IEEET ransactions on Pattern Analysis and Machine Intelligence(T-PAMI, September).

9 2013 International Conference on Learning Representations; 2013 ICML Workshop on Representation Learning Challenges; 2013 ICML Workshop on Deep Learning for Audio, Speech, andLanguage Processing; 2013 ICASSP Special Session on New Types of Deep Neural Net-work Learning for Speech Recognition and Related authors have been actively involved in deep Learning research andin organizing or providing several of the above events, tutorials, andeditorials. In particular, they gave tutorials and invited lectures onthis topic at various places. Part of this monograph is based on theirtutorials and lecture embarking on describing details of deep Learning , let s pro-vide necessary definitions. Deep Learning has various closely relateddefinitions or high-level descriptions: Definition 1: A class of machine Learning techniques thatexploit many layers of non-linear information processing for200 Introductionsupervised or unsupervised feature extraction and transforma-tion, and for pattern analysis and classification.

10 Definition 2: A sub-field within machine Learning that is basedon algorithms for Learning multiple levels of representation inorder to model complex relationships among data. Higher-levelfeatures and concepts are thus defined in terms of lower-levelones, and such a hierarchy of features is called a deep architec-ture. Most of these models are based on unsupervised Learning ofrepresentations. (Wikipedia on Deep Learning around March2012.) Definition 3: A sub-field of machine Learning that is basedon Learning several levels of representations, corresponding to ahierarchy of features or factors or concepts, where higher-levelconcepts are defined from lower-level ones, and the same lower-level concepts can help to define many higher-level concepts. Deeplearning is part of a broader family of machine Learning methodsbased on Learning representations. An observation ( , an image)can be represented in many ways ( , a vector of pixels), butsome representations make it easier to learn tasks of interest ( ,is this the image of a human face?)