Lecture 15 Symmetric matrices, quadratic forms, matrix ...

1 EE263 Autumn 2007-08 Stephen BoydLecture 15 Symmetric matrices, quadratic forms, matrixnorm, and SVD eigenvectors of Symmetric matrices quadratic forms inequalities for quadratic forms positive semidefinite matrices norm of a matrix singular value decomposition15 1 Eigenvalues of Symmetric matricessupposeA Rn nis Symmetric , ,A=ATfact:the eigenvalues ofAare realto see this, supposeAv= v,v6= 0,v CnthenvTAv=vT(Av) = vTv= nXi=1|vi|2but alsovTAv=(Av)Tv=( v)Tv= nXi=1|vi|2so we have = , , R(hence, can assumev Rn) Symmetric matrices, quadratic forms, matrix norm, and SVD15 2 Eigenvectors of Symmetric matricesfact:there is a set of orthonormal eigenvectors ofA, ,q1, .. , iqi,qTiqj= ijin matrix form: there is an 1AQ=QTAQ= hence we can expressAasA=Q QT=nXi=1 iqiqTiin particular,qiare both left and right eigenvectorsSymmetric matrices, quadratic forms, matrix norm, and SVD15 3 InterpretationsA=Q QTreplacementsxQTx QTxAxQTQ linear mappingy=Axcan be decomposed as resolve intoqicoordinates scale coordinates by i reconstitute with basisqiSymmetric matrices, quadratic forms, matrix norm, and SVD15 4or, geometrically, rotate byQT diagonal real scale ( dilation ) by rotate back byQdecompositionA=nXi=1 iqiqTiexpressesAas linear combination of 1-dimensional projectionsSymmetric matrices, quadratic forms, matrix norm, and SVD15 5example.

2 A= 1/2 3/23/2 1/2 = 1 2 1 11 1 1 00 2 1 2 1 11 1 Txq1q2q1qT1xq2qT2x 2q2qT2x 1q1qT1xAxSymmetric matrices, quadratic forms, matrix norm, and SVD15 6proof(case of idistinct)supposev1, .. , vnis a set of linearly independent eigenvectors ofA:Avi= ivi,kvik= 1then we havevTi(Avj) = jvTivj= (Avi)Tvj= ivTivjso( i j)vTivj= 0fori6=j, i6= j, hencevTivj= 0 in this case we can say: eigenvectorsareorthogonal in general case ( inot distinct) we must say: eigenvectorscan bechosento be orthogonalSymmetric matrices, quadratic forms, matrix norm, and SVD15 7 Example: RC circuitv1vnc1cni1inresistive circuitck vk= ik,i=GvG=GT Rn nis conductance matrix of resistive circuitthus v= C 1 GvwhereC=diag(c1, .. , cn)note C 1 Gis not symmetricSymmetric matrices, quadratic forms, matrix norm, and SVD15 8use statexi= civi, so x=C1/2 v= C 1/2GC 1/2xwhereC1/2=diag( c1.)

3 , cn)we conclude: eigenvalues 1, .. , nof C 1/2GC 1/2(hence, C 1G) are real eigenvectorsqi(inxicoordinates) can be chosen orthogonal eigenvectors in voltage coordinates,si=C 1/2qi, satisfy C 1 Gsi= isi,sTiCsi= ijSymmetric matrices, quadratic forms, matrix norm, and SVD15 9 quadratic formsa functionf:Rn Rof the formf(x) =xTAx=nXi,j=1 Aijxixjis called aquadratic formin a quadratic form we may as well assumeA=ATsincexTAx=xT((A+AT)/2)x((A+AT) /2is called thesymmetric partofA)uniqueness:ifxTAx=xTBxfor allx RnandA=AT,B=BT, thenA=BSymmetric matrices, quadratic forms, matrix norm, and SVD15 10 Examples kBxk2=xTBTBx Pn 1i=1(xi+1 xi)2 kF xk2 kGxk2sets defined by quadratic forms: {x|f(x) =a}is called aquadratic surface {x|f(x) a}is called aquadratic regionSymmetric matrices, quadratic forms, matrix norm, and SVD15 11 Inequalities for quadratic formssupposeA=AT,A=Q QTwith eigenvalues sorted so 1 nxTAx=xTQ QTx= (QTx)T (QTx)=nXi=1 i(qTix)2 1nXi=1(qTix)2= , we havexTAx 1xTxSymmetric matrices, quadratic forms, matrix norm, and SVD15 12similar argument showsxTAx nkxk2, so we have nxTx xTAx 1xTxsometimes 1is called max, nis called minnote also thatqT1Aq1= 1kq1k2,qTnAqn= nkqnk2,so the inequalities are tightSymmetric matrices, quadratic forms, matrix norm, and SVD15 13 Positive semidefinite and positive definite matricessupposeA=AT Rn nwe sayAispositive semidefiniteifxTAx 0for allx denotedA 0(and sometimesA 0) A 0if and only if min(A)

4 0, , all eigenvalues are nonnegative notthe same asAij 0for alli, jwe sayAispositive definiteifxTAx >0for allx6= 0 denotedA >0 A >0if and only if min(A)>0, , all eigenvalues are positiveSymmetric matrices, quadratic forms, matrix norm, and SVD15 14 matrix inequalities we sayAisnegative semidefiniteif A 0 we sayAisnegative definiteif A >0 otherwise, we sayAisindefinitematrix inequality: ifB=BT Rnwe sayA BifA B 0,A < BifB A >0, example: A 0meansAis positive semidefinite A > BmeansxTAx > xTBxfor allx6= 0 Symmetric matrices, quadratic forms, matrix norm, and SVD15 15many properties that you d guess hold actually do, , ifA BandC D, thenA+C B+D ifB 0thenA+B A ifA 0and 0, then A 0 A2 0 ifA >0, thenA 1>0matrix inequality is only apartial order: we can haveA6 B,B6 A(such matrices are calledincomparable) Symmetric matrices, quadratic forms, matrix norm, and SVD15 16 EllipsoidsifA=AT>0, the setE={x|xTAx 1}is anellipsoidinRn, centered at0s1s2 ESymmetric matrices, quadratic forms, matrix norm, and SVD15 17semi-axes are given bysi= 1/2iqi, : eigenvectors determine directions of semiaxes eigenvalues determine lengths of semiaxesnote.

5 In directionq1,xTAxislarge, hence ellipsoid isthinin directionq1 in directionqn,xTAxissmall, hence ellipsoid isfatin directionqn p max/ mingives maximumeccentricityif E={x|xTBx 1}, whereB >0, thenE E A BSymmetric matrices, quadratic forms, matrix norm, and SVD15 18 Gain of a matrix in a directionsupposeA Rm n(not necessarily square or Symmetric )forx Rn,kAxk/kxkgives theamplification factororgainofAin thedirectionxobviously, gain varies with direction of inputxquestions: what is maximum gain ofA(and corresponding maximum gain direction)? what is minimum gain ofA(and corresponding minimum gain direction)? how does gain ofAvary with direction? Symmetric matrices, quadratic forms, matrix norm, and SVD15 19 matrix normthe maximum gainmaxx6=0kAxkkxkis called thematrix normorspectral normofAand is denotedkAkmaxx6=0kAxk2kxk2= maxx6=0xTATA xkxk2= max(ATA)so we havekAk=p max(ATA)similarly the minimum gain is given byminx6=0kAxk/kxk=q min(ATA) Symmetric matrices, quadratic forms, matrix norm, and SVD15 20note that ATA Rn nis Symmetric andATA 0so min, max 0 max gain input direction isx=q1, eigenvector ofATAassociatedwith max min gain input direction isx=qn, eigenvector ofATAassociated with minSymmetric matrices, quadratic forms, matrix norm, and SVD15 21example:A= 1 23 45 6 ATA= 35 4444 56 = TthenkAk=p max(ATA) =.

6 = 1, A = = matrices, quadratic forms, matrix norm, and SVD15 22min gain isp min(ATA) = : = 1, A = = allx6= 0, we kAxkkxk matrices, quadratic forms, matrix norm, and SVD15 23 Properties of matrix norm consistent with vector norm: matrix norm ofa Rn 1isp max(aTa) = aTa for anyx,kAxk kAkkxk scaling:kaAk=|a|kAk triangle inequality:kA+Bk kAk+kBk definiteness:kAk= 0 A= 0 norm of product:kABk kAkkBkSymmetric matrices, quadratic forms, matrix norm, and SVD15 24 Singular value decompositionmore complete picture of gain properties ofAgiven bysingular valuedecomposition(SVD) ofA:A=U VTwhere A Rm n,Rank(A) =r U Rm r,UTU=I V Rn r,VTV=I =diag( 1, .. , r), where 1 r>0 Symmetric matrices, quadratic forms, matrix norm, and SVD15 25withU= [u1 ur],V= [v1 vr],A=U VT=rXi=1 iuivTi iare the (nonzero)singular valuesofA viare therightorinput singular vectorsofA uiare theleftoroutput singular vectorsofASymmetric matrices, quadratic forms, matrix norm, and SVD15 26 ATA= (U VT)T(U VT) =V 2 VThence: viare eigenvectors ofATA(corresponding to nonzero eigenvalues) i=p i(ATA)(and i(ATA) = 0fori > r) kAk= 1 Symmetric matrices, quadratic forms, matrix norm, and SVD15 27similarly,AAT= (U VT)(U VT)T=U 2 UThence: uiare eigenvectors ofAAT(corresponding to nonzero eigenvalues) i=p i(AAT)(and i(AAT) = 0fori > r) u1.

7 Urare orthonormal basis forrange(A) v1, .. vrare orthonormal basis forN(A) Symmetric matrices, quadratic forms, matrix norm, and SVD15 28 InterpretationsA=U VT=rXi=1 iuivTixVTx VTxAxVTU linear mappingy=Axcan be decomposed as compute coefficients ofxalong input directionsv1, .. , vr scale coefficients by i reconstitute along output directionsu1, .. , urdifference with eigenvalue decomposition for symmetricA: input andoutput directions aredifferentSymmetric matrices, quadratic forms, matrix norm, and SVD15 29 v1is most sensitive (highest gain) input direction u1is highest gain output direction Av1= 1u1 Symmetric matrices, quadratic forms, matrix norm, and SVD15 30 SVD gives clearer picture of gain as function of input/output directionsexample:considerA R4 4with =diag(10,7, , ) input components along directionsv1andv2are amplified (by about10) and come out mostly along plane spanned byu1,u2 input components along directionsv3andv4are attenuated (by about10) kAxk/kxkcan range Ais nonsingular for some applications you might sayAiseffectivelyrank 2 Symmetric matrices, quadratic forms, matrix norm, and SVD15 31example.

8 A R2 2, with 1= 1, 2= resolvexalongv1, v2:vT1x= ,vT2x= , ,x= + now formAx= (vT1x) 1u1+ (vT2x) 2u2= ( )(1)u1+ ( )( )u2v1v2u1u2xAxSymmetric matrices, quadratic forms, matrix norm, and SVD15 32