The Matrix Cookbook

1 The Matrix Cookbook [ ]Kaare Brandt PetersenMichael Syskind PedersenVersion: November 14, 2008 What is this?These pages are a collection of facts (identities, approxima-tions, inequalities, relations, ..) about matrices and matters relating to is collected in this form for the convenience of anyone who wants a quickdesktop reference .Disclaimer:The identities, approximations and relations presented here wereobviously not invented but collected, borrowed and copied from a large amountof sources. These sources include similar but shorter notes found on the internetand appendices in books - see the references for a full :Very likely there are errors, typos, and mistakes for which we apolo-gize and would be grateful to receive corrections at ongoing:The project of keeping a large repository of relations involvingmatrices is naturally ongoing and the version will be apparent from the date inthe :Your suggestion for additional content or elaboration of sometopics is most welcome at algebra, Matrix relations, Matrix identities, derivative ofdeterminant, derivative of inverse Matrix , differentiate a :We would like to thank the following for contributionsand suggestions: Bill Baxter, Brian Templeton, Christian Rish j, ChristianSchr oppel Douglas L.

2 Theobald, Esben Hoegh-Rasmussen, Glynne Casteel, JanLarsen, Jun Bin Gao, J urgen Struckmeier, Kamil Dedecius, Korbinian Strim-mer, Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, LoicThibaut, Miguel Bar ao, Ole Winther, Pavel Sakov, Stephan Hattinger, VasileSima, Vincent Rabaud, Zhaoshui He. We would also like thank The OticonFoundation for funding our PhD Trace and Determinants .. The Special Case 2x2 .. 52 Derivatives of a Determinant .. Derivatives of an Inverse .. Derivatives of Eigenvalues .. Derivatives of Matrices, Vectors and Scalar Forms .. Derivatives of Traces .. Derivatives of vector norms .. Derivatives of Matrix norms .. Derivatives of Structured Matrices .. 143 Basic .. Exact Relations .. Implication on Inverses .. Approximations .. Generalized Inverse .. Pseudo Inverse .. 204 Complex Complex Derivatives.

3 Higher order and non-linear derivatives .. Inverse of complex sum .. 265 Solutions and Solutions to linear equations .. Eigenvalues and Eigenvectors .. Singular Value Decomposition .. Triangular Decomposition .. LU decomposition .. LDM decomposition .. LDL decompositions .. 326 Statistics and Definition of Moments .. Expectation of Linear Combinations .. Weighted Scalar Variable .. 35 Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page 2 CONTENTSCONTENTS7 Multivariate Cauchy .. Dirichlet .. Normal .. Normal-Inverse Gamma .. Gaussian .. Multinomial .. Student s t .. Wishart .. Wishart, Inverse .. 388 Basics .. Moments .. Miscellaneous .. Mixture of Gaussians .. 449 Special Block matrices .. Discrete Fourier Transform Matrix , The .. Hermitian Matrices and skew-Hermitian.

4 Idempotent Matrices .. Orthogonal matrices .. Positive Definite and Semi-definite Matrices .. Singleentry Matrix , The .. Symmetric, Skew-symmetric/Antisymmetric .. Toeplitz Matrices .. Transition matrices .. Units, Permutation and Shift .. Vandermonde Matrices .. 5710 Functions and Functions and Series .. Kronecker and Vec Operator .. Vector Norms .. Matrix Norms .. Rank .. Integral Involving Dirac Delta Functions .. Miscellaneous .. 63A One-dimensional Gaussian .. One Dimensional Mixture of Gaussians .. 65B Proofs and Misc Proofs .. 67 Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page 3 CONTENTSCONTENTSN otation and NomenclatureAMatrixAijMatrix indexed for some purposeAiMatrix indexed for some purposeAijMatrix indexed for some purposeAnMatrix indexed for some purposeorThe power of a square matrixA 1 The inverse Matrix of the matrixAA+The pseudo inverse Matrix of the matrixA(see Sec.)

5 A1/2 The square root of a Matrix (if unique), not elementwise(A)ijThe (i, j).th entry of the matrixAAijThe (i, j).th entry of the matrixA[A]ijTheij-submatrix, row and column deletedaVectoraiVector indexed for some purposeaiThe element of the vectoraaScalar<zReal part of a scalar<zReal part of a vector<ZReal part of a Matrix =zImaginary part of a scalar=zImaginary part of a vector=ZImaginary part of a matrixdet(A) Determinant ofATr(A) Trace of the matrixAdiag(A) Diagonal Matrix of the matrixA, (diag(A))ij= ijAijeig(A) Eigenvalues of the matrixAvec(A) The vector-version of the matrixA(see Sec. )supSupremum of a set||A|| Matrix norm (subscript if any denotes what norm)ATTransposed matrixA TThe inverse of the transposed and vice versa,A T= (A 1)T= (AT) Complex conjugated matrixAHTransposed and complex conjugated Matrix (Hermitian)A BHadamard (elementwise) productA BKronecker product0 The null Matrix .

6 Zero in all identity matrixJijThe single-entry Matrix , 1 at (i, j) and zero elsewhere A positive definite Matrix A diagonal matrixPetersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page 41 BASICS1 Basics(AB) 1=B 1A 1(1)( ) 1=..C 1B 1A 1(2)(AT) 1= (A 1)T(3)(A+B)T=AT+BT(4)(AB)T=BTAT(5)( )T=..CTBTAT(6)(AH) 1= (A 1)H(7)(A+B)H=AH+BH(8)(AB)H=BHAH(9)( )H=..CHBHAH(10) Trace and DeterminantsTr(A) = iAii(11)Tr(A) = i i, i= eig(A)(12)Tr(A) = Tr(AT)(13)Tr(AB) = Tr(BA)(14)Tr(A+B) = Tr(A) + Tr(B)(15)Tr(ABC) = Tr(BCA) = Tr(CAB)(16)det(A) = i i i= eig(A)(17)det(cA) =cndet(A),ifA Rn n(18)det(AT) = det(A)(19)det(AB) = det(A) det(B)(20)det(A 1) = 1/det(A)(21)det(An) = det(A)n(22)det(I+uvT) = 1 +uTv(23)det(I+ A) =1 + Tr(A), small(24) The Special Case 2x2 Consider the matrixAA=[A11A12A21A22]Determinant and tracedet(A) =A11A22 A12A21(25)Tr(A) =A11+A22(26)Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page The Special Case 2x21 BASICSE igenvalues 2 Tr(A) + det(A) = 0 1=Tr(A) + Tr(A)2 4 det(A)2 2=Tr(A) Tr(A)2 4 det(A)2 1+ 2= Tr(A) 1 2= det(A)Eigenvectorsv1 [A12 1 A11]v2 [A12 2 A11]InverseA 1=1det(A)[A22 A12 A21A11](27)Petersen & Pedersen, The Matrix Cookbook , Version.

7 November 14, 2008,Page 62 DERIVATIVES2 DerivativesThis section is covering differentiation of a number of expressions with respect toa matrixX. Note that it is always assumed thatXhasno special structure, the elements ofXare independent ( not symmetric, Toeplitz, positivedefinite). See section for differentiation of structured matrices. The basicassumptions can be written in a formula as Xkl Xij= ik lj(28)that is for vector forms,[ x y]i= xi y[ x y]i= x yi[ x y]ij= xi yjThe following rules are general and very useful when deriving the differential ofan expression ([19]): A= 0(Ais a constant) (29) ( X) = X(30) (X+Y) = X+ Y(31) (Tr(X)) = Tr( X)(32) (XY) = ( X)Y+X( Y)(33) (X Y) = ( X) Y+X ( Y)(34) (X Y) = ( X) Y+X ( Y)(35) (X 1) = X 1( X)X 1(36) (det(X)) = det(X)Tr(X 1 X)(37) (ln(det(X))) = Tr(X 1 X)(38) XT= ( X)T(39) XH= ( X)H(40) Derivatives of a General form det(Y) x= det(Y)Tr[Y 1 Y x](41) det(Y) x x= det(Y)[Tr[Y 1 Y x x]+Tr[Y 1 Y x]Tr[Y 1 Y x] Tr[(Y 1 Y x) (Y 1 Y x)]](42)Petersen & Pedersen, The Matrix Cookbook , Version.

8 November 14, 2008,Page Derivatives of an Inverse2 Linear forms det(X) X= det(X)(X 1)T(43) k det(X) XikXjk= ijdet(X)(44) det(AXB) X= det(AXB)(X 1)T= det(AXB)(XT) 1(45) Square formsIfXis square and invertible, then det(XTAX) X= 2 det(XTAX)X T(46)IfXis not square butAis symmetric, then det(XTAX) X= 2 det(XTAX)AX(XTAX) 1(47)IfXis not square andAis not symmetric, then det(XTAX) X= det(XTAX)(AX(XTAX) 1+ATX(XTATX) 1) (48) Other nonlinear formsSome special cases are (See [9, 7]) ln det(XTX)| X= 2(X+)T(49) ln det(XTX) X+= 2XT(50) ln|det(X)| X= (X 1)T= (XT) 1(51) det(Xk) X=kdet(Xk)X T(52) Derivatives of an InverseFrom [27] we have the basic identity Y 1 x= Y 1 Y xY 1(53)Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page Derivatives of Eigenvalues2 DERIVATIVES from which it follows (X 1)kl Xij= (X 1)ki(X 1)jl(54) aTX 1b X= X TabTX T(55) det(X 1) X= det(X 1)(X 1)T(56) Tr(AX 1B) X= (X 1 BAX 1)T(57) Tr((X+A) 1) X= ((X+A) 1(X+A) 1)T(58)From [32] we have the following result: LetAbe ann ninvertible squarematrix,Wbe the inverse ofA, andJ(A) is ann n-variate and differentiablefunction with respect toA, then the partial differentials ofJwith respect toAandWsatisfy J A= A T J WA Derivatives of Eigenvalues X eig(X) = XTr(X) =I(59) X eig(X) = Xdet(X) = det(X)X T(60) Derivatives of Matrices, Vectors and Scalar First Order xTa x= aTx x=a(61) aTXb X=abT(62) aTXTb X=baT(63) aTXa X= aTXTa X=aaT(64) X Xij=Jij(65) (XA)ij Xmn= im(A)nj= (JmnA)ij(66) (XTA)ij Xmn= in(A)mj= (JnmA)ij(67)Petersen & Pedersen, The Matrix Cookbook , Version.

9 November 14, 2008,Page Derivatives of Matrices, Vectors and Scalar Forms 2 Second Order Xij klmnXklXmn= 2 klXkl(68) bTXTXc X=X(bcT+cbT)(69) (Bx+b)TC(Dx+d) x=BTC(Dx+d) +DTCT(Bx+b)(70) (XTBX)kl Xij= lj(XTB)ki+ kj(BX)il(71) (XTBX) Xij=XTBJij+JjiBX(Jij)kl= ik jl(72)See Sec for useful properties of the Single-entry matrixJij xTBx x= (B+BT)x(73) bTXTDXc X=DTXbcT+DXcbT(74) X(Xb+c)TD(Xb+c) = (D+DT)(Xb+c)bT(75)AssumeWis symmetric, then s(x As)TW(x As) = 2 ATW(x As)(76) x(x s)TW(x s) = 2W(x s)(77) s(x s)TW(x s) = 2W(x s)(78) x(x As)TW(x As) = 2W(x As)(79) A(x As)TW(x As) = 2W(x As)sT(80)As a case with complex values the following holds (a xHb)2 x= 2b(a xHb) (81)This formula is also known from the LMS algorithm [14] Higher order and non-linear (Xn)kl Xij=n 1 r=0(XrJijXn 1 r)kl(82)Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page Derivatives of Traces2 DERIVATIVESFor proof of the above, see XaTXnb=n 1 r=0(Xr)TabT(Xn 1 r)T(83) XaT(Xn)TXnb=n 1 r=0[Xn 1 rabT(Xn)TXr+(Xr)TXnabT(Xn 1 r)T](84)See for a functions ofx, (x),r=r(x), and thatAis aconstant, then xsTAr=[ s x]TAr+[ r x]TATs(85) x(Ax)T(Ax)(Bx)T(Bx)= xxTATAxxTBTBx(86)= 2 ATAxxTBBx 2xTATAxBTBx(xTBTBx)2(87) Gradient and HessianUsing the above we have for the gradient and the Hessianf=xTAx+bTx(88) xf= f x= (A+AT)x+b(89) 2f x xT=A+AT(90) Derivatives of TracesAssumeF(X) to be a differentiable function of each of the elements ofX.

10 Itthen holds that Tr(F(X)) X=f(X)Twheref( ) is the scalar derivative ofF( ).Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page Derivatives of Traces2 First Order XTr(X) =I(91) XTr(XA) =AT(92) XTr(AXB) =ATBT(93) XTr(AXTB) =BA(94) XTr(XTA) =A(95) XTr(AXT) =A(96) XTr(A X) = Tr(A)I(97) Second Order XTr(X2) = 2XT(98) XTr(X2B) = (XB+BX)T(99) XTr(XTBX) =BX+BTX(100) XTr(XBXT) =XBT+XB(101) XTr(AXBX) =ATXTBT+BTXTAT(102) XTr(XTX) = 2X(103) XTr(BXXT) = (B+BT)X(104) XTr(BTXTCXB) =CTXBBT+CXBBT(105) XTr[XTBXC]=BXC+BTXCT(106) XTr(AXBXTC) =ATCTXBT+CAXB(107) XTr[(AXB+C)(AXC+C)T]= 2AT(AXB+C)BT(108) XTr(X X) = XTr(X)Tr(X) = 2Tr(X)I(109)See [7].Petersen & Pedersen, The Matrix Cookbook , Version: November 14, 2008,Page Derivatives of vector norms2 Higher Order XTr(Xk) =k(Xk 1)T(110) XTr(AXk) =k 1 r=0(XrAXk r 1)T(111) XTr[BTXTCXXTCXB]=CXXTCXBBT+CTXBBTXTCTX+C XBBTXTCX+CTXXTCTXBBT(112) Other XTr(AX 1B) = (X 1 BAX 1)T= X TATBTX T(113)AssumeBandCto be symmetric, then XTr[(XTCX) 1A]= (CX(XTCX) 1)(A+AT)(XTCX) 1(114) XTr[(XTCX) 1(XTBX)]= 2CX(XTCX) 1 XTBX(XTCX) 1+2BX(XTCX) 1(115) XTr[(A+XTCX) 1(XTBX)]= 2CX(A+XTCX) 1 XTBX(A+XTCX) 1+2BX(A+XTCX) 1(116)See [7].