Transcription of The Matrix Cookbook
1 The Matrix Cookbook [ ]Kaare Brandt PetersenMichael Syskind PedersenVersion: November 15, 20121 IntroductionWhat is this?These pages are a collection of facts (identities, approxima-tions, inequalities, relations, ..) about matrices and matters relating to is collected in this form for the convenience of anyone who wants a quickdesktop reference .Disclaimer:The identities, approximations and relations presented here wereobviously not invented but collected, borrowed and copied from a large amountof sources. These sources include similar but shorter notes found on the internetand appendices in books - see the references for a full :Very likely there are errors, typos, and mistakes for which we apolo-gize and would be grateful to receive corrections at ongoing:The project of keeping a large repository of relations involvingmatrices is naturally ongoing and the version will be apparent from the date inthe :Your suggestion for additional content or elaboration of sometopics is most welcome algebra, Matrix relations, Matrix identities, derivative ofdeterminant, derivative of inverse Matrix , differentiate a.
2 We would like to thank the following for contributionsand suggestions: Bill Baxter, Brian Templeton, Christian Rish j, ChristianSchr oppel, Dan Boley, Douglas L. Theobald, Esben Hoegh-Rasmussen, EvripidisKarseras, Georg Martius, Glynne Casteel, Jan Larsen, Jun Bin Gao, J urgenStruckmeier, Kamil Dedecius, Karim T. Abou-Moustafa, Korbinian Strimmer,Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, Loic Thibaut,Markus Froeb, Michael Hubatka, Miguel Bar ao, Ole Winther, Pavel Sakov,Stephan Hattinger, Troels Pedersen, Vasile Sima, Vincent Rabaud, ZhaoshuiHe. We would also like thank The Oticon Foundation for funding our & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page 2 CONTENTSCONTENTSC ontents1 Trace.
3 Determinant .. The Special Case 2x2 ..72 Derivatives of a Determinant .. Derivatives of an Inverse .. Derivatives of Eigenvalues .. Derivatives of Matrices, Vectors and Scalar Forms .. Derivatives of Traces .. Derivatives of vector norms .. Derivatives of Matrix norms .. Derivatives of Structured Matrices .. 143 Basic .. Exact Relations .. Implication on Inverses .. Approximations .. Generalized Inverse .. Pseudo Inverse .. 214 Complex Complex Derivatives .. Higher order and non-linear derivatives .. Inverse of complex sum.
4 275 Solutions and Solutions to linear equations .. Eigenvalues and Eigenvectors .. Singular Value Decomposition .. Triangular Decomposition .. LU decomposition .. LDM decomposition .. LDL decompositions .. 336 Statistics and Definition of Moments .. Expectation of Linear Combinations .. Weighted Scalar Variable .. 367 Multivariate Cauchy .. Dirichlet .. Normal .. Normal-Inverse Gamma .. Gaussian .. Multinomial .. 37 Petersen & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page Student s t.
5 Wishart .. Wishart, Inverse .. 398 Basics .. Moments .. Miscellaneous .. Mixture of Gaussians .. 449 Special Block matrices .. Discrete Fourier Transform Matrix , The .. Hermitian Matrices and skew-Hermitian .. Idempotent Matrices .. Orthogonal matrices .. Positive Definite and Semi-definite Matrices .. Singleentry Matrix , The .. Symmetric, Skew-symmetric/Antisymmetric .. Toeplitz Matrices .. Transition matrices .. Units, Permutation and Shift .. Vandermonde Matrices .. 5710 Functions and Functions and Series.
6 Kronecker and Vec Operator .. Vector Norms .. Matrix Norms .. Rank .. Integral Involving Dirac Delta Functions .. Miscellaneous .. 63A One-dimensional Gaussian .. One Dimensional Mixture of Gaussians .. 65B Proofs and Misc Proofs .. 66 Petersen & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page 4 CONTENTSCONTENTSN otation and NomenclatureAMatrixAijMatrix indexed for some purposeAiMatrix indexed for some purposeAijMatrix indexed for some purposeAnMatrix indexed for some purposeorThe power of a square matrixA 1 The inverse Matrix of the matrixAA+The pseudo inverse Matrix of the matrixA(see Sec.)
7 A1/2 The square root of a Matrix (if unique), not elementwise(A)ijThe (i,j).th entry of the matrixAAijThe (i,j).th entry of the matrixA[A]ijTheij-submatrix, row and column deletedaVector (column-vector)aiVector indexed for some purposeaiThe element of the vectoraaScalar<zReal part of a scalar<zReal part of a vector<ZReal part of a Matrix =zImaginary part of a scalar=zImaginary part of a vector=ZImaginary part of a matrixdet(A)Determinant ofATr(A)Trace of the matrixAdiag(A)Diagonal Matrix of the matrixA, (diag(A))ij= ijAijeig(A)Eigenvalues of the matrixAvec(A)The vector-version of the matrixA(see Sec. )supSupremum of a set||A|| Matrix norm (subscript if any denotes what norm)ATTransposed matrixA TThe inverse of the transposed and vice versa,A T= (A 1)T= (AT) Complex conjugated matrixAHTransposed and complex conjugated Matrix (Hermitian)A BHadamard (elementwise) productA BKronecker product0 The null Matrix .
8 Zero in all identity matrixJijThe single-entry Matrix , 1 at (i,j) and zero elsewhere A positive definite Matrix A diagonal matrixPetersen & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page 51 BASICS1 Basics(AB) 1=B 1A 1(1)( ) 1=..C 1B 1A 1(2)(AT) 1= (A 1)T(3)(A+B)T=AT+BT(4)(AB)T=BTAT(5)( )T=..CTBTAT(6)(AH) 1= (A 1)H(7)(A+B)H=AH+BH(8)(AB)H=BHAH(9)( )H=..CHBHAH(10) TraceTr(A) = iAii(11)Tr(A) = i i, i= eig(A)(12)Tr(A) = Tr(AT)(13)Tr(AB) = Tr(BA)(14)Tr(A+B) = Tr(A) + Tr(B)(15)Tr(ABC) = Tr(BCA) = Tr(CAB)(16)aTa= Tr(aaT)(17) DeterminantLetAbe ann (A) = i i i= eig(A)(18)det(cA) =cndet(A),ifA Rn n(19)det(AT) = det(A)(20)det(AB) = det(A) det(B)(21)det(A 1) = 1/det(A)(22)det(An) = det(A)n(23)det(I+uvT) = 1 +uTv(24)Forn= 2:det(I+A) = 1 + det(A) + Tr(A)(25)Forn= 3:det(I+A) = 1 + det(A) + Tr(A) +12Tr(A)2 12Tr(A2)(26)Petersen & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page The Special Case 2x21 BASICSForn= 4.
9 Det(I+A) = 1 + det(A) + Tr(A) +12+Tr(A)2 12Tr(A2)+16Tr(A)3 12Tr(A)Tr(A2) +13Tr(A3)(27)For small , the following approximation holdsdet(I+ A) =1 + det(A) + Tr(A) +12 2Tr(A)2 12 2Tr(A2)(28) The Special Case 2x2 Consider the matrixAA=[A11A12A21A22]Determinant and tracedet(A) =A11A22 A12A21(29)Tr(A) =A11+A22(30)Eigenvalues 2 Tr(A) + det(A) = 0 1=Tr(A) + Tr(A)2 4 det(A)2 2=Tr(A) Tr(A)2 4 det(A)2 1+ 2= Tr(A) 1 2= det(A)Eigenvectorsv1 [A12 1 A11]v2 [A12 2 A11]InverseA 1=1det(A)[A22 A12 A21A11](31)Petersen & Pedersen, The Matrix Cookbook , Version: November 15, 2012,Page 72 DERIVATIVES2 DerivativesThis section is covering differentiation of a number of expressions with respect toa matrixX.
10 Note that it is always assumed thatXhasno special structure, the elements ofXare independent ( not symmetric, Toeplitz, positivedefinite). See section for differentiation of structured matrices. The basicassumptions can be written in a formula as Xkl Xij= ik lj(32)that is for vector forms,[ x y]i= xi y[ x y]i= x yi[ x y]ij= xi yjThe following rules are general and very useful when deriving the differential ofan expression ([19]): A= 0(Ais a constant)(33) ( X) = X(34) (X+Y) = X+ Y(35) (Tr(X)) = Tr( X)(36) (XY) = ( X)Y+X( Y)(37) (X Y) = ( X) Y+X ( Y)(38) (X Y) = ( X) Y+X ( Y)(39) (X 1) = X 1( X)X 1(40) (det(X)) = Tr(adj(X) X)(41) (det(X)) = det(X)Tr(X 1 X)(42) (ln(det(X))) = Tr(X 1 X)(43) XT= ( X)T(44) XH= ( X)H(45) Derivatives of a General form det(Y) x= det(Y)Tr[Y 1 Y x](46) k det(X) XikXjk= ijdet(X)(47) 2det(Y) x2= det(Y)[Tr[Y 1 Y x x]+Tr[Y 1 Y x]Tr[Y 1 Y x] Tr[(Y 1 Y x)(Y 1 Y x)]](48)Petersen & Pedersen, The Matrix Cookbook , Version.
