Mathematics for Machine Learning - Assets

1 Cambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressMathematics for Machine LearningThe fundamental mathematical tools needed to understand Machine Learning includelinear algebra, analytic geometry, matrix decompositions, vector calculus, optimiza-tion, probability, and statistics. These topics are traditionally taught in disparatecourses, making it hard for data science or computer science students, or profes-sionals, to efficiently learn the self-contained textbook bridges the gap between mathematical and machinelearning texts, introducing the mathematical concepts with a minimum of prerequi-sites. It uses these concepts to derive four central Machine Learning methods: linearregression, principal component analysis, Gaussian mixture models, and supportvector machines. For students and others with a mathematical background, thesederivations provide a starting point to Machine Learning texts.

2 For those learningthe Mathematics for the first time, the methods help build intuition and practicalexperience with applying mathematical chapter includes worked examples and exercises to test understanding. Pro-gramming tutorials are offered on the book s web Peter Deisenrothis the DeepMind Chair in Artificial Intelligence at Uni-versity College London. Prior to this, Marc was a faculty member at Imperial CollegeLondon. His research areas include data-efficient Learning , probabilistic modeling,and autonomous decision making. His research received Best Paper Awards at theICRA 2014 and the ICCAS 2016. Marc has been awarded the President s Award forOutstanding Early Career Researcher at Imperial College London, a Google FacultyResearch Award, and a Microsoft PhD Aldo Faisalleads the Brain & Behaviour Lab at Imperial College London,where he is faculty at the Departments of Bioengineering and Computing and aFellow of the Data Science Institute. He is the director of the 20 Mio UnitedKingdom Research and Innovation (UKRI) Center for Doctoral Training in AIfor Healthcare.

3 He obtained a PhD in computational neuroscience at CambridgeUniversity and became Junior Research Fellow in the Computational and BiologicalLearning Lab. His research is at the interface of neuroscience and Machine learningto understand and reverse engineer brains and Soon Ongis Principal Research Scientist at the Machine LearningResearch Group, Data61, CSIRO and Adjunct Associate Professor at the AustralianNational University. His research focuses on enabling scientific discovery byextending statistical Machine Learning methods. He received his PhD in computerscience at Australian National University in 2005. He has been a lecturer in theDepartment of Computer Science at ETH Z urich, and has worked in the DiagnosticGenomics Team at NICTA in University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressCambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A.

4 Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressMathematics for MachineLearningMarc Peter DeisenrothUniversity College LondonA. Aldo FaisalImperial College LondonCheng Soon OngData61, CSIROC ambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressUniversity Printing House, Cambridge CB2 8BS, United KingdomOne Liberty Plaza, 20th Floor, New York, NY 10006, USA477 Williamstown Road, Port Melbourne, VIC 3207, Australia314 321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi 110025, India79 Anson Road, #06 04/06, Singapore 079906 Cambridge University Press is part of the University of furthers the University s mission by disseminating knowledge in the pursuit ofeducation, Learning , and research at the highest international levels of on this title: : Marc Peter Deisenroth, A.

5 Aldo Faisal, and Cheng Soon Ong 2020 This publication is in copyright. Subject to statutory exceptionand to the provisions of relevant collective licensing agreements,no reproduction of any part may take place without the writtenpermission of Cambridge University published 2020 Printed in Singapore by Markono Print Media Pte LtdA catalogue record for this publication is available from the British of Congress Cataloging-in-Publication DataNames: Deisenroth, Marc Peter, author.|Faisal, A. Aldo, author.|Ong, Cheng Soon, : Mathematics for Machine Learning / Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon : Cambridge ; New York, NY : Cambridge University Press, 2020.|Includes bibliographical references and : LCCN 2019040762 (print)|LCCN 2019040763 (ebook)|ISBN 9781108470049 (hardback)|ISBN 9781108455145 (paperback)|ISBN 9781108679930 (epub)Subjects: LCSH: Machine Learning : LCC .D45 2020 (print)|LCC (ebook)|DDC dc23LC record available at ebook record available at 978-1-108-47004-9 HardbackISBN 978-1-108-45514-5 PaperbackAdditional resources for this publication at University Press has no responsibility for the persistence or accuracyof URLs for external or third-party internet websites referred to in this publicationand does not guarantee that any content on such websites is, or will remain,accurate or University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A.

6 Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressContentsList of SymbolsixPrefacexiAcknowledgmentsxvPart I Mathematical Foundations1 Introduction and Finding Words for Two Ways to Read This Exercises and Feedback72 Linear Systems of Linear Solving Systems of Linear Vector Linear Basis and Linear Affine Further Reading50 Exercises513 Analytic inner Lengths and Angles and Orthonormal Orthogonal inner product of Orthogonal Further Reading79 Exercises80vCambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressviContents4 Matrix Determinant and Eigenvalues and Cholesky Eigendecomposition and Singular Value Matrix Matrix Further Reading116 Exercises1185 Vector Differentiation of Univariate Partial Differentiation and Gradients of Vector-Valued Gradients of Useful Identities for Computing Backpropagation and Automatic Higher-Order Linearization and Multivariate Taylor Further Reading149 Exercises1506 Probability and Construction of a Probability Discrete and Continuous Sum Rule, product Rule.

7 And Bayes Summary Statistics and Gaussian Conjugacy and the Exponential Change of Variables/Inverse Further Reading197 Exercises1987 Continuous Optimization Using Gradient Constrained Optimization and Lagrange Convex Further Reading220 Exercises221 Part II Central Machine Learning Problems8 When Models Meet Data, Models, and Empirical Risk Minimization232 Cambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University Parameter Probabilistic Modeling and Directed Graphical Model Selection2549 Linear Problem Parameter Bayesian Linear Maximum Likelihood as Orthogonal Further Reading28310 Dimensionality Reduction with Principal Component Problem Maximum Variance Projection Eigenvector Computation and Low-Rank PCA in High Key Steps of PCA in Latent Variable Further Reading31011 Density Estimation with Gaussian Mixture Gaussian Mixture Parameter Learning via Maximum EM Latent-Variable Further Reading33212 Classification with Support Vector Separating Primal Support Vector Dual Support Vector Numerical Further Reading355 References357 Index367 Cambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A.

8 Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressCambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressList of SymbolsSymbolTypical meaninga, b, c, , , Scalars are lowercasex,y,zVectors are bold lowercaseA,B,CMatrices are bold uppercasex ,A Transpose of a vector or matrixA 1 Inverse of a matrix x,y inner product ofxandyx yDot product ofxandyB=(b1,b2,b3)(Ordered) tupleB=[b1,b2,b3]Matrix of column vectors stacked horizontallyB={b1,b2,b3}Set of vectors (unordered)Z,NIntegers and natural numbers, respectivelyR,CReal and complex numbers, respectivelyRnn-dimensional vector space of real numbers xUniversal quantifier: for allx xExistential quantifier: there existsxa:=bais defined asba=:bbis defined asaa bais proportional tob, ,a=constant bg fFunction composition: gafterf If and only if= ImpliesA,CSetsa Aais an element of the setA Empty setDNumber of dimensions; indexed byd=1.

9 ,DNNumber of data points; indexed byn=1,..,NImIdentity matrix of sizem m0m,nMatrix of zeros of sizem n1m,nMatrix of ones of sizem neiStandard/canonical vector (whereiis thecomponent that is1)dim(V)Dimensionality of vector space VixCambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressxList of SymbolsSymbolTypical meaningrk(A)Rank of matrixAIm( )Image of linear mapping ker( )Kernel (null space ) of a linear mapping span[b1]Span (generating set) ofb1tr(A)Trace ofAdet(A)Determinant ofA| |Absolute value or determinant (depending on context) Norm; Euclidean unless specified Eigenvalue or Lagrange multiplierE Eigenspace corresponding to eigenvalue Parameter vector f xPartial derivative offwith respect toxdfdxTotal derivative offwith respect tox GradientLLagrangianLNegative log-likelihood(nk)Binomial coefficient,nchoosekVX[x]Variance ofxwith respect to the random variableXEX[x]Expectation ofxwith respect to the random variableXCovX,Y[x,y]Covariance Y|ZXis conditionally independent ofYgivenZX pRandom variableXis distributed according topN( , )Gaussian distribution with mean and covariance Ber( )Bernoulli distribution with parameter Bin(N, )Binomial distribution with parametersN, Beta( , )Beta distribution with parameters , List of Abbreviations and gratia (Latin: for example)GMMG aussian mixture est (Latin.)

10 This means) , identically distributedMAPM aximum a posterioriMLEM aximum likelihood estimation/estimatorONBO rthonormal basisPCAP rincipal component analysisPPCAP robabilistic principal component analysisREFRow-echelon formSPDS ymmetric, positive definiteSVMS upport vector machineCambridge University Press978-1-108-47004-9 Mathematics for Machine LearningMarc Peter Deisenroth , A. Aldo Faisal , Cheng Soon Ong FrontmatterMore in this web service Cambridge University PressPrefaceMachine Learning is the latest in a long line of attempts to distill humanknowledge and reasoning into a form that is suitable for constructing machinesand engineering automated systems. As Machine Learning becomes moreubiquitous and its software packages become easier to use, it is natural anddesirable that the low-level technical details are abstracted away and hiddenfrom the practitioner. However, this brings with it the danger that a practitionerbecomes unaware of the design decisions and, hence, the limits of machinelearning enthusiastic practitioner who is interested to learn more about the magicbehind successful Machine Learning algorithms currently faces a daunting set ofprerequisite knowledge: Programming languages and data analysis tools Large-scale computation and the associated frameworks Mathematics and statistics and how Machine Learning builds on itAt universities, introductory courses on Machine Learning tend to spend earlyparts of the course covering some of these prerequisites.