Properties of the Trace and Matrix Derivatives

1 Properties of the Trace and Matrix DerivativesJohn DuchiContents1 Notation12 Matrix multiplication13 Gradient of linear function14 Derivative in a trace25 Derivative of product in trace26 Derivative of function of a matrix37 Derivative of linear transformed input to function38 Funky Trace derivative39 Symmetric Matrices and Eigenvectors41 NotationA few things on notation (which may not be very consistent, actually): The columns of a matrixA Rm narea1throughan, while the rows are given (as vectors) by aT1throught Matrix multiplicationFirst, consider a matrixA Rn n. We have thatAAT=n i=1aiaTi,that is, that the product ofAATis the sum of the outer products of the columns ofA. To see this,consider that(AAT)ij=n p=1apiapjbecause thei, jelement is theithrow ofA, which is the vector a1i, a2i, , ani , dotted with thejthcolumn ofAT, which is a1j, , we look at the matrixAAT, we see thatAAT= np=1ap1ap1 np= np=1apnap1 np=1apnapn =n i=1 ai1ai1 ainain =n i=1aiaTi3 Gradient of linear functionConsiderAx, whereA Rm nandx Rn.

2 We have xAx= x aT1x x x aTmx =[ a1 a2 am]=ATNow let us considerxTAxforA Rn n,x Rn. We have thatxTAx=xT[ aT1x aT2x aTnx]T=x1 aT1x+ +xn aTnxIf we take the derivative with respect to one of thexls, we have thelcomponent for each ai, whichis to sayail, and the term forxl aTlx, which gives us that xlxTAx=n i=1xiail+ aTlx=aTlx+ the end, we see that xxTAx=Ax+ Derivative in a traceRecall (as inOld and New Matrix Algebra Useful for Statistics) that we can define the differentialof a functionf(x) to be the part off(x+dx) f(x) that is linear indx, is a constant timesdx. Then, for example, for a vector valued functionf, we can havef(x+dx) =f(x) +f (x)dx+ (higher order terms).In the above,f is the derivative (or Jacobian). Note that the gradient is the transpose of an arbitrary matrixA.

3 We see thattr(AdX)dX=tr aTndxn dX= ni=1 , we have[tr(AdX)dX]ij=[ ni=1 aTidxi xji]=aijso thattr(AdX)dX= that this is the Jacobian Derivative of product in traceIn this section, we prove that AtrAB=BTtrAB= tr ~a1 ~a2 .. ~an ~b1~b2 ~bn = tr ~a1T~b1~a1T~b2 ~a1T~bn~a2T~b1~a2T~b2 ~a2T~ ~anT~b1~anT~b2 ~anT~bn =m i=1a1ibi1+m i=1a2ibi2+..+m i=1anibin trAB aij=bji AtrAB=BT6 Derivative of function of a matrixHere we prove that ATf(A) = ( Af(A))T. ATf(A) = f(A) A11 f(A) A21 f(A) An1 f(A) A12 f(A) A22 f(A) f(A) A1n f(A) A2n f(A) Ann = ( Af(A))T7 Derivative of linear transformed input to functionConsider a functionf:Rn R. Suppose we have a matrixA Rn mand a vectorx Rm. Wewish to compute xf(Ax). By the chain rule, we have f(Ax) xi=n k=1 f(Ax) (Ax)k (Ax)k xi=n k=1 f(Ax) (Ax)k ( aTkx) xi=n k=1 f(Ax) (Ax)k aki=n k=1 kf(Ax)aki=aTi f(Ax).

4 3As such, xf(Ax) =AT f(Ax). Now, if we would like to get the second derivative of this function(third Derivatives would be a little nice, but I do not like tensors), we have 2f(Ax) xi xj= xjaTi f(Ax) = xjn k=1aki f(Ax) (Ax)k=n l=1n k=1aki 2f(Ax) (Ax)k (Ax)lali=aTi 2f(Ax)ajFrom this, it is easy to see that 2xf(Ax) =AT 2f(Ax) Funky Trace derivativeIn this section, we prove that AtrABATC=CAB+ this bit, let us haveAB=f(A), wherefis Matrix -valued. AtrABATC= Atrf(A)ATC= trf( )ATC+ trf(A) TC= (ATC)Tf ( ) + ( Ttrf(A) TC)T=CTABT+ ( Ttr TCf(A))T=CTABT+ ((Cf(A))T)T=CTABT+CAB9 Symmetric Matrices and EigenvectorsIn this we prove that for a symmetric matrixA Rn n, all the eigenvalues are real, and that theeigenvectors ofAform an orthonormal basis , we prove that the eigenvalues are real.

5 Suppose one is complex: we have xTx= (Ax)Tx=xTATx=xTAx= , all the eigenvalues are , we suppose we have at least one eigenvectorv6= 0 ofA. Consider a spaceWof vectorsorthogonal tov. We then have that, forw W,(Aw)Tv=wTATv=wTAv= wTv= , we have a set of vectorsWthat, when transformed byA, are still orthogonal tov, so ifwe have an original eigenvectorvofA, then a simple inductive argument shows that there is anorthonormal set of see that there is at least one eigenvector, consider the characteristic polynomial ofA:X(A) = det(A I).The field is algebraicly closed, so there is at least one complex rootr, so we have thatA rIissingular and there is a vectorv6= 0 that is an eigenvector ofA. Thusris a real eigenvalue, so wehave the base case for our induction, and the proof is