Lecture 15 Symmetric matrices, quadratic forms, matrix ...

1 EE263 Autumn 2007-08 Stephen BoydLecture 15 Symmetric matrices, quadratic forms, matrixnorm, and SVD eigenvectors of Symmetric matrices quadratic forms inequalities for quadratic forms positive semidefinite matrices norm of a matrix singular value decomposition15 1 Eigenvalues of Symmetric matricessupposeA Rn nis Symmetric , ,A=ATfact:the eigenvalues ofAare realto see this, supposeAv= v,v6= 0,v CnthenvTAv=vT(Av) = vTv= nXi=1|vi|2but alsovTAv=(Av)Tv=( v)Tv= nXi=1|vi|2so we have = , , R(hence, can assumev Rn) Symmetric matrices, quadratic forms, matrix norm, and SVD15 2 Eigenvectors of Symmetric matricesfact:there is a set of orthonormal eigenvectors ofA, ,q1, .. , iqi,qTiqj= ijin matrix form: there is an 1AQ=QTAQ= hence we can expressAasA=Q QT=nXi=1 iqiqTiin particular,qiare both left and right eigenvectorsSymmetric matrices, quadratic forms, matrix norm, and SVD15 3 InterpretationsA=Q QTreplacementsxQTx QTxAxQTQ linear mappingy=Axcan be decomposed as resolve intoqicoordinates scale coordinates by i reconstitute with basisqiSymmetric matrices, quadratic forms, matrix norm, and SVD15 4or, geometrically, rotate byQT diagonal real scale ( dilation ) by rotate back byQdecompositionA=nXi=1 iqiqTiexpressesAas linear combination of 1-dimensional projectionsSymmetric matrices, quadratic forms, matrix norm, and SVD15 5example:A= 1/2 3/23/2 1/2 = 1 2 1 11 1 1 00 2 1 2 1 11 1 Txq1q2q1qT1xq2qT2x 2q2qT2x 1q1qT1xAxSymmetric matrices, quadratic forms, matrix norm, and SVD15 6proof(case of idistinct)supposev1.

2 , vnis a set of linearly independent eigenvectors ofA:Avi= ivi,kvik= 1then we havevTi(Avj) = jvTivj= (Avi)Tvj= ivTivjso( i j)vTivj= 0fori6=j, i6= j, hencevTivj= 0 in this case we can say: eigenvectorsareorthogonal in general case ( inot distinct) we must say: eigenvectorscan bechosento be orthogonalSymmetric matrices, quadratic forms, matrix norm, and SVD15 7 Example: RC circuitv1vnc1cni1inresistive circuitck vk= ik,i=GvG=GT Rn nis conductance matrix of resistive circuitthus v= C 1 GvwhereC=diag(c1, .. , cn)note C 1 Gis not symmetricSymmetric matrices, quadratic forms, matrix norm, and SVD15 8use statexi= civi, so x=C1/2 v= C 1/2GC 1/2xwhereC1/2=diag( c1, .. , cn)we conclude: eigenvalues 1, .. , nof C 1/2GC 1/2(hence, C 1G) are real eigenvectorsqi(inxicoordinates) can be chosen orthogonal eigenvectors in voltage coordinates,si=C 1/2qi, satisfy C 1 Gsi= isi,sTiCsi= ijSymmetric matrices, quadratic forms, matrix norm, and SVD15 9 quadratic formsa functionf:Rn Rof the formf(x) =xTAx=nXi,j=1 Aijxixjis called aquadratic formin a quadratic form we may as well assumeA=ATsincexTAx=xT((A+AT)/2)x((A+AT) /2is called thesymmetric partofA)uniqueness:ifxTAx=xTBxfor allx RnandA=AT,B=BT, thenA=BSymmetric matrices, quadratic forms, matrix norm, and SVD15 10 Examples kBxk2=xTBTBx Pn 1i=1(xi+1 xi)2 kF xk2 kGxk2sets defined by quadratic forms.

3 {x|f(x) =a}is called aquadratic surface {x|f(x) a}is called aquadratic regionSymmetric matrices, quadratic forms, matrix norm, and SVD15 11 Inequalities for quadratic formssupposeA=AT,A=Q QTwith eigenvalues sorted so 1 nxTAx=xTQ QTx= (QTx)T (QTx)=nXi=1 i(qTix)2 1nXi=1(qTix)2= , we havexTAx 1xTxSymmetric matrices, quadratic forms, matrix norm, and SVD15 12similar argument showsxTAx nkxk2, so we have nxTx xTAx 1xTxsometimes 1is called max, nis called minnote also thatqT1Aq1= 1kq1k2,qTnAqn= nkqnk2,so the inequalities are tightSymmetric matrices, quadratic forms, matrix norm, and SVD15 13 Positive semidefinite and positive definite matricessupposeA=AT Rn nwe sayAispositive semidefiniteifxTAx 0for allx denotedA 0(and sometimesA 0) A 0if and only if min(A) 0, , all eigenvalues are nonnegative notthe same asAij 0for alli, jwe sayAispositive definiteifxTAx >0for allx6= 0 denotedA >0 A >0if and only if min(A)>0, , all eigenvalues are positiveSymmetric matrices, quadratic forms, matrix norm, and SVD15 14 matrix inequalities we sayAisnegative semidefiniteif A 0 we sayAisnegative definiteif A >0 otherwise, we sayAisindefinitematrix inequality: ifB=BT Rnwe sayA BifA B 0,A < BifB A >0, example: A 0meansAis positive semidefinite A > BmeansxTAx > xTBxfor allx6= 0 Symmetric matrices, quadratic forms, matrix norm, and SVD15 15many properties that you d guess hold actually do, , ifA BandC D, thenA+C B+D ifB 0thenA+B A ifA 0and 0, then A 0 A2 0 ifA >0, thenA 1>0matrix inequality is only apartial order.

4 We can haveA6 B,B6 A(such matrices are calledincomparable) Symmetric matrices, quadratic forms, matrix norm, and SVD15 16 EllipsoidsifA=AT>0, the setE={x|xTAx 1}is anellipsoidinRn, centered at0s1s2 ESymmetric matrices, quadratic forms, matrix norm, and SVD15 17semi-axes are given bysi= 1/2iqi, : eigenvectors determine directions of semiaxes eigenvalues determine lengths of semiaxesnote: in directionq1,xTAxislarge, hence ellipsoid isthinin directionq1 in directionqn,xTAxissmall, hence ellipsoid isfatin directionqn p max/ mingives maximumeccentricityif E={x|xTBx 1}, whereB >0, thenE E A BSymmetric matrices, quadratic forms, matrix norm, and SVD15 18 Gain of a matrix in a directionsupposeA Rm n(not necessarily square or Symmetric )forx Rn,kAxk/kxkgives theamplification factororgainofAin thedirectionxobviously, gain varies with direction of inputxquestions: what is maximum gain ofA(and corresponding maximum gain direction)?

5 What is minimum gain ofA(and corresponding minimum gain direction)? how does gain ofAvary with direction? Symmetric matrices, quadratic forms, matrix norm, and SVD15 19 matrix normthe maximum gainmaxx6=0kAxkkxkis called thematrix normorspectral normofAand is denotedkAkmaxx6=0kAxk2kxk2= maxx6=0xTATA xkxk2= max(ATA)so we havekAk=p max(ATA)similarly the minimum gain is given byminx6=0kAxk/kxk=q min(ATA) Symmetric matrices, quadratic forms, matrix norm, and SVD15 20note that ATA Rn nis Symmetric andATA 0so min, max 0 max gain input direction isx=q1, eigenvector ofATAassociatedwith max min gain input direction isx=qn, eigenvector ofATAassociated with minSymmetric matrices, quadratic forms, matrix norm, and SVD15 21example:A= 1 23 45 6 ATA= 35 4444 56 = TthenkAk=p max(ATA) = : = 1, A = = matrices, quadratic forms, matrix norm, and SVD15 22min gain isp min(ATA) =.

6 = 1, A = = allx6= 0, we kAxkkxk matrices, quadratic forms, matrix norm, and SVD15 23 Properties of matrix norm consistent with vector norm: matrix norm ofa Rn 1isp max(aTa) = aTa for anyx,kAxk kAkkxk scaling:kaAk=|a|kAk triangle inequality:kA+Bk kAk+kBk definiteness:kAk= 0 A= 0 norm of product:kABk kAkkBkSymmetric matrices, quadratic forms, matrix norm, and SVD15 24 Singular value decompositionmore complete picture of gain properties ofAgiven bysingular valuedecomposition(SVD) ofA:A=U VTwhere A Rm n,Rank(A) =r U Rm r,UTU=I V Rn r,VTV=I =diag( 1, .. , r), where 1 r>0 Symmetric matrices, quadratic forms, matrix norm, and SVD15 25withU= [u1 ur],V= [v1 vr],A=U VT=rXi=1 iuivTi iare the (nonzero)singular valuesofA viare therightorinput singular vectorsofA uiare theleftoroutput singular vectorsofASymmetric matrices, quadratic forms, matrix norm, and SVD15 26 ATA= (U VT)T(U VT) =V 2 VThence: viare eigenvectors ofATA(corresponding to nonzero eigenvalues) i=p i(ATA)(and i(ATA) = 0fori > r) kAk= 1 Symmetric matrices, quadratic forms, matrix norm, and SVD15 27similarly,AAT= (U VT)(U VT)T=U 2 UThence: uiare eigenvectors ofAAT(corresponding to nonzero eigenvalues) i=p i(AAT)(and i(AAT) = 0fori > r) u1.

7 Urare orthonormal basis forrange(A) v1, .. vrare orthonormal basis forN(A) Symmetric matrices, quadratic forms, matrix norm, and SVD15 28 InterpretationsA=U VT=rXi=1 iuivTixVTx VTxAxVTU linear mappingy=Axcan be decomposed as compute coefficients ofxalong input directionsv1, .. , vr scale coefficients by i reconstitute along output directionsu1, .. , urdifference with eigenvalue decomposition for symmetricA: input andoutput directions aredifferentSymmetric matrices, quadratic forms, matrix norm, and SVD15 29 v1is most sensitive (highest gain) input direction u1is highest gain output direction Av1= 1u1 Symmetric matrices, quadratic forms, matrix norm, and SVD15 30 SVD gives clearer picture of gain as function of input/output directionsexample:considerA R4 4with =diag(10,7, , ) input components along directionsv1andv2are amplified (by about10) and come out mostly along plane spanned byu1,u2 input components along directionsv3andv4are attenuated (by about10) kAxk/kxkcan range Ais nonsingular for some applications you might sayAiseffectivelyrank 2 Symmetric matrices, quadratic forms, matrix norm, and SVD15 31example:A R2 2, with 1= 1, 2= resolvexalongv1, v2.

8 VT1x= ,vT2x= , ,x= + now formAx= (vT1x) 1u1+ (vT2x) 2u2= ( )(1)u1+ ( )( )u2v1v2u1u2xAxSymmetric matrices, quadratic forms, matrix norm, and SVD15 32