Lecture 1. Random vectors and multivariate normal …

1 Lecture 1. Random vectors and multivariate normal Moments of Random vectorA Random vectorXof sizepis a column vector consisting ofprandom variablesX1,..,Xpand isX= (X1,..,Xp) . The mean or expectation ofXis defined by the vector ofexpectations, E(X) = E(X1)..E(Xp) ,which exists ifE|Xi|< for alli= 1,.., a Random vector of sizepandYbe a Random vector of sizeq. Forany non- Random matricesA(m p),B(m q),C(1 n), andD(m n),E(AX+BY) =AE(X) +BE(Y),E(AXC+D) =AE(X)C+ a Random vectorXof sizepsatisfyingE(X2i)< for alli= 1,..,p, the variance covariance matrix (or just covariance matrix) ofXis Cov(X) =E[(X EX)(X EX) ].

2 The covariance matrix ofXis ap psquare, symmetric matrix. In particular, ij=Cov(Xi,Xj) = Cov(Xj,Xi) = properties:1. Cov(X) =E(XX ) E(X)E(X) .2. Ifc=c(p 1)is a constant, Cov(X+c) = Cov(X).3. IfA(m p)is a constant, Cov(AX) =ACov(X)A .Lemma pmatrix is a covariance matrix if and only if it is multivariate normal distribution - nonsingular caseRecall that the univariate normal distribution with mean and variance 2has densityf(x) = (2 2) 12exp[ 12(x ) 2(x )].Similarly, the multivariate normal distribution for the special case of nonsingular covariancematrix is defined as Rpand (p p)>0.

3 A Random vectorX Rphasp-variate normaldistribution with mean and covariance matrix if it has probability density functionf(x) =|2 | 12exp[ 12(x ) 1(x )],(1)forx Rp. We use the notationX Np( , ).Theorem Np( , )for >0, 12(X ) Np(0,Ip), 12Y+ whereY Np(0,Ip), (X) = and Cov(X) = ,4. for any fixedv Rp,v Xis univariate (X ) 1(X ) 2(p).Example1 (Bivariate normal ). Geometry of multivariate normalThe multivariate normal distribution has location parameter and the shape parameter >0. In particular, let s look into the contour of equal densityEc={x Rp:f(x) =c0}={x Rp: (x ) 1(x ) =c2}.

4 Moreover, consider the spectral decomposition of =U U whereU= [u1,..,up] and = diag( 1,.., p) with 1 2 .. p>0. TheEc, for anyc >0, is an ellipsoidcentered around with principal axesuiof length proportional to i. If =Ip, theellipsoid is the surface of a sphere of radiusccentered at .As an example, consider a bivariate normal distributionN2(0, ) with =[2 11 2]=[cos( /4) sin( /4)sin( /4)cos( /4)][3 00 1][cos( /4) sin( /4)sin( /4)cos( /4)] .The location of the distribution is the origin ( =0), and the shape ( ) of the distributionis determined by the ellipse given by the two principal axes (one at 45 degree line, theother at -45 degree line).

5 Figure 1 shows the density function and the correspondingEcforc= ,1, ,2,..2 Figure 1: Bivariate normal density and its contours. Notice that an ellipses in the plane canrepresent a bivariate normal distribution . In higher dimensionsd >2, ellipsoids play thesimilar General multivariate normal distributionThe characteristic function of a Random vectorXis defined as X(t) =E(eit X),fort that the characteristic function isC-valued, and always exists. We collect someimportant X(t) = Y(t) if and only ifXL= IfXandYare independent, then X+Y(t) = X(t) Y(t). Xif and only if Xn(t) X(t) for important corollary follows from the uniqueness of the characteristic 4(Cramer Wold device).

6 IfXis ap 1random vector then its distribution isuniquely determined by the distributions of linear functions oft X, for everyt 4 paves the way to the definition of (general) multivariate normal Random vectorX Rphas a multivariate normal distribution ift Xis anunivariate normal for allt definition says thatXis MVN if every projection ofXonto a 1-dimensional subspaceis normal , with a convention that a degenerate distribution chas a normal distribution withvariance 0, ,c N(c,0). The definition does not require that Cov(X) is characteristic function of a multivariate normal distribution with mean and covariance matrix 0is, fort Rp, (t) = exp[it 12t t].

7 If >0, then the pdf exists and is the same as (1).In the following, the notationX N( , ) is valid for a non-negative definite . How-ever, whenever 1appears in the statement, is assumed to be positive Np( , )andY=AX+bforA(q p)andb(q 1), thenY Nq(A +b,A A ).Next two results are concerning independence and conditional distributions of normalrandom vectors . LetX1andX2be the partition ofXwhose dimensions arerands,r+s=p, and suppose and are partitioned accordingly. That is,X=[X1X2] Np([ 1 2],[ 11 12 21 22]).Proposition normal Random vectorsX1andX2are independent if and only ifCov(X1,X2) = 12= conditional distribution ofX1givenX2=x2isNr( 1+ 12 122(x2 2), 11 12 122 21) new Random vectorsX 1=X1 12 122X2andX 2=X2,X =[X 1X 2]=AX,A=[Ir 12 1220(s r)Is].

8 By Proposition 6,X is multivariate normal . An inspection of the covariance matrix ofX leads thatX 1andX 2are independent. The result follows by writingX1=X 1+ 12 122X2,and that the distribution (law) ofX1givenX2=x2isL(X1|X2=x2) =L(X 1+ 12 122X2|X2=x2) =L(X 1+ 12 122x2|X2=x2), which is a MVN of multivariate Central Limit TheoremIfX1,X2,.. Rpare withE(Xi) = and Cov(X) = , thenn 12n j=1(Xj ) Np(0, )asn ,or equivalently,n12( Xn ) Np(0, )asn ,where Xn=12 nj= delta-method can be used for asymptotic normality ofh( Xn) for some functionh:Rp R. In particular, denote h(x) for the gradient ofhatx.

9 Using the first twoterms of Taylor series,h( Xn) =h( ) + ( h( )) ( Xn ) +Op( Xn 22),Then Slutsky s theorem gives the result, n(h( Xn) h( )) = ( h( )) n( Xn ) +Op( n( Xn ) ( Xn )) ( h( )) Np(0, )asn ,=Np(0,( h( )) ( h( ))) Quadratic forms in normal Random vectorsLetX Np( , ). A quadratic form inXis a Random variable of the formY=X AX=p i=1p j=1 XiaijXj,whereAis ap psymmetric matrix andXiis theith element ofX. We are interested inthe distribution of quadratic forms and the conditions under which two quadratic forms special case: IfX Np(0,Ip) andA=Ip,Y=X AX=X X=p i=1X2i 2(p).

10 The following:1. Ap pmatrixAis idempotent ifA2= IfAis symmetric, thenA= , where = diag( i) and is IfAis symmetric idempotent,5(a) its eigenvalues are either 0 or 1,(b) rank(A) = #{non zero eigenvalues}= trace(A).Theorem Np(0, 2I)andAbe ap psymmetric matrix. ThenY=X AX 2 2(m)if and only ifAis idempotent of rankm < Np(0, )andAbe ap psymmetric matrix. ThenY=X AX 2(m)if and only if either i)A is idempotent of rankmor ii) Ais idempotent of Np( , ) then (X ) 1(X ) 2(p).Theorem Np(0,I)andAbe ap psymmetric matrix, andBbe ak pmatrix. IfBA= 0, thenBXandX AXare N( , 2) The sample mean Xnand the sample varianceS2n=(n 1) 1 ni=1(Xi Xn)2are independent.