The Bivariate Normal Distribution - IIT Kanpur

1 The Bivariate Normal DistributionThis is Section of the 1st edition (2002) of the book Introduc-tion to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. Thematerial in this section was not included in the 2nd edition (2008).LetUandVbe two independent Normal random variables, and consider twonew random variablesXandYof the formX=aU+bV,Y=cU+dV,wherea, b, c, d, are some scalars. Each one of the random variablesXandYisnormal, since it is a linear function of independent Normal random variables. Furthermore, becauseXandYare linear functions of the same two independentnormal random variables, their joint PDF takes a special form, known as thebi-variate normalPDF. The Bivariate Normal PDF has several useful and elegantproperties and, for this reason, it is a commonly employed model. In this section,we derive many such properties, both qualitative and analytical, culminating ina closed-form expression for the joint PDF.

2 To keep the discussion simple, werestrict ourselves to the case whereXandYhave zero Normal Random VariablesTwo random variablesXandYare said to bejointly normalif they canbeexpressedintheformX=aU+bV,Y=cU+dV,w hereUandVare independent Normal random that ifXandYare jointly Normal , then any linear combinationZ=s1X+s2Y For the purposes of this section, we adopt the following convention. A randomvariable which is always equal to a constant will also be called Normal , with zerovariance, even though it does not have a PDF. With this convention, the family ofnormal random variables is closed under linear operations. That is, ifXis Normal ,thenaX+bis also Normal , even ifa= Bivariate Normal Distributionhas a Normal Distribution . The reason is that if we haveX=aU+bVandY=cU+dVfor some independent Normal random variablesUandV,thenZ=s1(aU+bV)+s2(cU+dV) =(as1+cs2)U+(bs1+ds2) ,Zis the sum of the independent Normal random variables (as1+cs2)Uand (bs1+ds2)V, and is therefore very important property of jointly Normal random variables, and whichwill be the starting point for our development, is that zero correlation Correlation Implies IndependenceIf two random variablesXandYare jointly Normal and are uncorrelated,then they are property can be verified using multivariate transforms, as thatUandVare independent zero-mean Normal random variables,and thatX=aU+bVandY=cU+dV,sothatXandYare jointly assume thatXandYare uncorrelated, and we wish to show that they areindependent.

3 Our first step is to derive a formula for the multivariate transformMX,Y(s1,s2) associated withXandY. Recall that ifZis a zero-mean normalrandom variable with variance 2Z, the associated transform isE[esZ]=MZ(s)=e 2Zs2/2,which implies thatE[eZ]=MZ(1) =e 2 us fix some scalarss1,s2,andletZ=s1X+s2Y. The random variableZis Normal , by our earlier discussion, with variance 2Z=s21 2X+s22 leads to the following formula for the multivariate transform associatedwith the uncorrelated pairXandY:MX,Y(s1,s2)=E[es1X+s2Y]=E[eZ]= e(s21 2X+s22 2Y) nowXandYbeindependentzero-mean Normal random variables withthe same variances 2 Xand 2 YasXandY, respectively. SinceXandYareindependent, they are also uncorrelated, and the preceding argument yieldsMX,Y(s1,s2)=e(s21 2X+s22 2Y) Bivariate Normal Distribution3 Thus, the two pairs of random variables (X, Y)and(X,Y) are associated withthe same multivariate transform. Since the multivariate transform completelydetermines the joint PDF, it follows that the pair (X, Y) has the same jointPDF as the pair (X,Y).

4 SinceXandYare independent,XandYmust alsobe independent, which establishes our Conditional Distribution ofXGivenYWe now turn to the problem of estimatingXgiven the value degenerate cases, we assume that bothXandYhave positivevariance. Let us define X= X YY, X=X X,where =E[XY] X Yis the correlation coefficient linear combinationsof independent Normal random variablesUandV, it follows thatYand Xarealso linear combinations ,Yand Xare jointly ,E[Y X]=E[YX] E[Y X]= X Y X Y 2Y= ,Yand Xare uncorrelated and, therefore, independent. Since Xis a scalarmultiple ofY, it follows that Xand Xare have so far decomposedXinto a sum of two independent Normal ran-dom variables, namely,X= X+ X= X YY+ take conditional expectations of both sides, givenY,toobtainE[X|Y]= X YE[Y|Y]+E[ X|Y]= X YY= X,where we have made use of the independence ofYand Xto setE[ X|Y]=0. Wehave therefore reached the important conclusion that the conditional expectationE[X|Y] is a linear function of the random the above decomposition, it is now easy to determine the conditionalPDF ofX.

5 Given a value ofY, the random variable X= XY/ Ybecomes Comparing with the formulas in the preceding section, it is seen that Xisdefined to be the linear least squares estimator ofX,and Xis the correspondingestimation error, although these facts are not needed for the argument that Bivariate Normal Distributiona known constant, but the Normal Distribution of the random variable Xisunaffected, since Xis independent ofY. Therefore, the conditional distributionofXgivenYis the same as the unconditional Distribution of X,shiftedby Xis Normal with mean zero and some variance 2 X, we conclude that theconditional Distribution ofXis also Normal with mean Xand the same variance 2 X. The variance of Xcan be found with the following calculation: 2 X=E[(X X YY)2]= 2X 2 X Y X Y+ 2 2X 2Y 2Y=(1 2) 2X,wherewehavemadeuseofthepropertyE[XY]= X summarize our conclusions below. Although our discussion used thezero-mean assumption, these conclusions also hold for the non-zero mean caseand we state them with this added generality; see the end-of-chapter of Jointly Normal Random VariablesLetXandYbe jointly Normal random variables.

6 XandYare independent if and only if they are uncorrelated. The conditional expectation ofXgivenYsatisfiesE[X|Y]=E[X]+ X Y(Y E[Y]).It is a linear function ofYand has a Normal PDF. The estimation error X=X E[X|Y] is zero-mean, Normal , andindependent ofY, with variance 2 X=(1 2) 2X. The conditional Distribution ofXgivenYis Normal with meanE[X|Y]and variance 2 determined the parameters of the PDF of Xand of the conditional PDFofX, we can give explicit formulas for these PDFs. We keep assuming thatThe Bivariate Normal Distribution5 XandYhave zero means and positive variances. Furthermore, to avoid thedegenerate where Xis identically zero, we assume that| |<1. We havef X( x)=f X|Y( x|y)=1 2 1 2 Xe x2/2 2 X,andfX|Y(x|y)=1 2 1 2 Xe (x X Yy)2/2 2 X,where 2 X=(1 2) ,fY(y)=1 2 Ye y2/2 2Y,and the multiplication rulefX,Y(x, y)=fY(y)fX|Y(x|y), we can obtain thejoint PDF ofXandY. This PDF is of the formfX,Y(x, y)=ce q(x,y),where the normalizing constant isc=12 1 2 X exponent termq(x, y) is a quadratic function ofxandy,q(x, y)=y22 2Y+(x X Yy)22(1 2) 2X,which after some straightforward algebra simplifies toq(x, y)=x2 2X 2 xy X Y+y2 2Y2(1 2).

7 An important observation here is thatthe joint PDF is completely deter-mined by X, Y,and .In the special case whereXandYare uncorrelated ( = 0), the joint PDFtakes the simple formfX,Y(x, y)=12 X Ye x22 2X y22 2Y,6 The Bivariate Normal Distributionwhich is just the product of two independent Normal PDFs. We can get someinsight into the form of this PDF by considering its contours, , sets of pointsat which the PDF takes a constant value. These contours are described by anequation of the formx2 2X+y2 2Y= constant,and are ellipses whose two axes are horizontal and the more general case whereXandYare dependent, a typical contouris described byx2 2X 2 xy X Y+y2 2Y= constant,and is again an ellipse, but its axes are no longer horizontal and vertical. illustrates the contours for two cases, one in which is positive and one inwhich is :Contours of the Bivariate Normal PDF. The diagram on the left(respectively, right) corresponds to a case of positive (respectively, negative) cor-relation coefficient.

8 Example thatXandZare zero-mean jointly Normal randomvariables, such that 2X=4, 2Z=17/9, andE[XZ] = 2. We define a new randomvariableY=2X 3Z. We wish to determine the PDF ofY, the conditional PDFofXgivenY, noted earlier, a linear function of two jointly Normal random variables isalso Normal . Thus,Yis Normal with variance 2Y=E[(2X 3Z)2]=4E[X2]+9E[Z2] 12E[XZ]=4 4+9 179 12 2= ,Yhas the Normal PDFfY(y)=1 2 3e y2 Bivariate Normal Distribution7We next note thatXandYare jointly Normal . The reason is thatXandZarelinear functions of two independent Normal random variables (by the definition ofjoint normality), so thatXandYare also linear functions of the same independentnormal random variables. The covariance ofXandYis equal toE[XY]=E[X(2X 3Z)]=2E[X2] 3E[XZ]=2 4 3 2= , the correlation coefficient ofXandY, denoted by ,isequalto =E[XY] X Y=22 3= conditional expectation ofXgivenYisE[X|Y]= X YY=13 23Y= conditional variance ofXgivenY(which is the same as the variance of X=X E[X|Y]) is 2 X=(1 2) 2X=(1 19)4=329,so that X= 32/3.

9 Hence, the conditional PDF ofXgivenYisfX|Y(x|y)=3 2 32e (x (2y/9))22 32 , the joint PDF ofXandYis obtained using either the multiplicationrulefX,Y(x, y)=fX(x)fX|Y(x|y), or by using the earlier developed formula forthe exponentq(x, y), and is equal tofX,Y(x, y)=12 32e y29+x24 23 xy2 32(1 (1/9)).We end with a cautionary note. IfXandYare jointly Normal , then eachrandom variableXandYis Normal . However, the converse is not true. Namely,if each of the random variablesXandYis Normal , it does not follow thatthey are jointly Normal , even if they are uncorrelated. This is illustrated in thefollowing a Normal Distribution with zero mean and unitvariance. LetZbe independent ofX,withP(Z=1)=P(Z= 1) = 1/2. Let8 The Bivariate Normal DistributionY=ZX, which is also Normal with zero mean. The reason is that conditionedon either value ofZ,Yhas the same Normal Distribution , hence its unconditionaldistribution is also Normal .

10 Furthermore,E[XY]=E[ZX2]=E[Z]E[X2]=0 1=0,soXandYare uncorrelated. On the other handXandYare clearly dependent.(For example, ifX=1,thenYmust be either 1or1.) IfXandYwere jointlynormal, we would have a contradiction to our earlier conclusion that zero correlationimplies independence. It follows thatXandYarenotjointly Normal , even thoughboth marginal distributions are Multivariate Normal PDFThe development in this section generalizes to the case of more than two randomvariables. For example, we can say that the random variablesX1,..,Xnarejointly Normal if all of them are linear functions of a setU1,..,Unof independentnormal random variables. We can then establish the natural extensions of theresults derived in this section. For example, it is still true that zero correlationimplies independence, that the conditional expectation of one random variablegiven some of the others is a linear function of the conditioning random variables,and that the conditional PDF ofX1.