Chapter 12 Multivariate normal distributions - Yale University

1 Page 1 Chapter 12 Multivariate normal distributionsThe Multivariate normal is the most useful, and most studied, of the standard joint dis-tributions in probability . A huge body of statistical theory depends on the properties of fam-ilies of random variables whose joint distribution is at least approximately Multivariate nor-mal. The bivariate case (two variables) is the easiest to understand, because it requires aminimum of notation; vector notation and matrix algebra becomes necessities when manyrandom variables are general bivariate normal is often used to model pairs of dependent random vari-ables, such as : the height and weight of an individual; or (as an approximation) the score astudent gets on a final exam and the total score she gets on the problem sets; or the heightsof father and son; and so on.

2 Many fancy statistical procedures implicitly require bivariate(or Multivariate , for more than two random variables) normalThe most general bivariate normal can be built from a pair of independent random vari-ables,XandY, each ;1/. For a constant with 1< <1, define randomvariablesUDXandVD XCp1 2 YThat is,.U; ;Y/AwhereAD 1 0p1 2 Notice thatEUDEVD0, 2 ; ;V/D ;X/Cp1 ;Y/D :Consequently, ; ;V/= From Chapter 10, the joint density ;V/is1jdetAjf .u;v/A 1 ; ;y/D12 exp x2Cy22 allx;yThe matrixAhas determinantp1 2and inverseA 1D p1 2 01 =p1 2 Statistics 241: 16 November 1997c David PollardChapter 12 Multivariate normal distributionsPage ; ;v/A ;v/A 1 ; ;v/ 1 0 .u;v/0=.1 2/Du2 2 uvCv21 2 ThusUandVhave joint density12 p1 2exp u2 2 2/ for allu;v:The joint distribution is sometimes called thestandard bivariate normaldistribution standard bivariate normalwith correlation.

3 The symmetry of inuandvimplies thatVhas the same marginal distribution asU,that is,Vis ;1/distributed. The calculation of the marginals densities involves thesame integration for both equals zero, the joint density factorizes into1p2 exp. u2=2/1p2 exp. v2=2/which implies independence ofUandV. That is, for random variables with a bivariate nor-mal distribution, zero correlation is equivalent to independence. The equivalence for bivari-ate normals probably accounts for the widespread confusion between the properties of in-dependence and zero correlation. In general, independence implies zero correlation, but notconversely.< > variablesSandTare said to have a bivariate normal distribution,with parametersESD S,ETD T, 2S, 2T, and correlation ifthe standardized random S/= T/= Thave a standard bivariatenormal distribution with correlation.

4 Problem shows how to calculate explicitly the joint density distributionsThe construction ofUandVfrom the independentXandYmakes the calculation of theconditional distribution ofVgivenUDua triviality: XCp1 2 YjXDxhas the distribution of xCp1 ;1/. That is,< >VjUDu N. u;1 2/The symmetry of the joint distribution ofUandVimplies thatUjVDv N. v;1 2/;a fact that you could check by explicit calculation of the ratio of joint to marginal densities: .u;v/.Z1 1 .u;v/duD1p2 p1 2exp .v 2/ < > the height (in inches) of a randomly chosen father, and letYde-note the height (in inches) of his son at maturity. Suppose each ofXandYhas aN. ; 2/distribution with D69 and D2. Suppose also thatXandYhave a bivariate normaldistribution with correlation Sam has a height of 74 inches, what would one predict about the ultimate height ofhis son Elmer?

5 Statistics 241: 16 November 1997c David PollardChapter 12 Multivariate normal distributionsPage 3In standardized units, /= DSam s standardized height, which happens to equal 2 /= DElmer s standardized ultimate height:By assumption, before the value ofUwas known, the ;V/has a standard bivariatenormal distribution with correlation . From the analog of formula< >,VjUD2:5 :5 ;1 2/In the original units,Elmer s heightjSam s height = 74 inches N. C2:5 ;.1 2/ 2 :5;3:64/Notice that Elmer s expected height (given that Sam is 74 inches) is less than his fa-ther s height. This fact is an example of a general phenomenon called regression towardsthe mean . The termregression, as a synonym for conditional expectation, has become regressioncommonplace in Statistics. Multivariate densitiesRandom variablesX1;X2;:::are said to have a jointly continuous distribution with jointdensity ;x2;:::; ;X2;:::;Xn/2 ;x2;::: ;x2;:::;xn/dx1dx2:::dxnfor each subsetAofRn.

6 The densityfmust be nonnegative and integrate to 1 is convenient to writeXfor therandom ;:::;Xn/, andxfor the random vectorgeneric ;:::;xn/inRn. Then the defining property for the joint density RnwhereR:::dxshould be understood as ann-fold integral.< > the random variablesX1;:::;Xnare independent, the joint density functionis equal to the product of the marginal densities for eachXi, and conversely. The proof issimilar to the proof for the bivariate example, if thefXigare independent and eachXihas ;1/distribution, thejoint density ;:::; /n=2exp Xi nx2i=2!for allx1;:::; /n=2exp. kxk2=2/for allxThe distribution is denoted ;In/. It is sometimes called the spherical normal distri-bution , because of the spherical symmetry of the density. The methods for finding joint densities for random variables defined as functions ofother random variables with jointly continuous distributions as explained over the last twoChapters extend to Multivariate distributions .

7 There is a problem with the drawing ofn-dimensional pictures, to keep track of the transformations, and one must remember to say n-dimensional volume instead of area, but otherwise calculations are not much more com-plicated than in two of coordinate axesThe spherical symmetry of the densityf. /is responsible for an important property of multi-variate normals. Letq1;:::;qnbe a new orthonormal basis forRn, and letZDW1q1C:::CWnqnStatistics 241: 16 November 1997c David PollardChapter 12 Multivariate normal distributionsPage 4be the representation forZin the new basis.< > ;:::;Wnare also ;1/distributed random two dimensions, the assertion follows from the transformation formulae of Chap-ter 10. If the axes are rorated through an angle , thenW1DZ1cos. /CZ2sin. /W2D Z1sin. /CZ2cos. /That is,.W1;W2 ;Z2/A whereA D cos.

8 / sin. /sin. /cos. / The matrixA has determinant 1 and inverseA . It is an orthogonal matrix; it preserveslengths. The joint density ;W2/is12 exp ;w2/A 1k2=2 D12 exp .w21Cw22/=2 z1z2w1w2 ball B (in Z-coordinates) = ball B* (in W-coordinates)A more intuitive explanation is based on the approximationPfZ2Bg (volume ofB)for a small ballBcentered atz. The transformation fromZtoWcorresponds to a rotation,soPfZ2 BgDPfW2B g;whereB is a ball of the same radius, but centered at the ;:::;wn/for whichw1q1C:::CwnqnDz. The last equality implieskwkDkzk, from which we getPfW2B g .2 / n=2exp. 12kwk2/(volume ofB ).That is,Whas the asserted spherical normal density.< > ;Z2;:::;Zn/have a spherical normal ;In/. Thechi-square, 2n, is defined as the distribution ofkZk2DZ21C:::CZ2n. chi-squareTo prove results about the spherical normal it is often merely a matter of transformingto an appropriate orthonormal basis.

9 < > ;Z2;:::;Znare independent, each ;1/. DefineNZDZ1C:::CZnnandTDXi NZ/2 Show thatNZhas ;1=n/distribution independently ofT, which has a 2n 241: 16 November 1997c David PollardChapter 12 Multivariate normal distributionsPage 5 Solution:Choose the new orthonormal basis ;1;:::;1/0=pn. Chooseq2;:::;qnhowever you like, provided they are orthogonal unit vectors, all orthogonal toq1. In the new coordinate system,ZDW1q1C:::CWnqnWe could calculate eachWiby dotting the sum on the right- hand side withqi: onlyWiwould survive. In particular,W1DZ q1DZ1C:::CZnpnDpnNZ:From Theorem< >we know thatW1has ;1/distribution. It follows thatNZhas ;1= random variableTequals the squared length of the NZ;:::;Zn NZ/DZ W1q1DW2q2C:::CWnqnThat is,TDkW2q2C:::CWnqnk2DW22C:::CW2n;a sum of squares ofn 1 ;1/random variables, which has a 2n , notice thatNZis a function ofW1, whereasTis a function of the independentrandom variablesW2;:::;Wn.

10 The independence ofNZandTfollows. Exercise for the reader:SupposeX1;:::Xnare independent, each distributedN. ; 2/.Apply the results from the last Exercise, /= , to deduce thatNXis dis-tributedN. ; 2=n/independently ofXi NX/2= 2;which has a 2n 241: 16 November 1997c David Pollar