More on Multivariate Gaussians

1 More on Multivariate GaussiansChuong B. DoNovember 21, 2008Up to this point in class, you have seen Multivariate Gaussians arisein a number of appli-cations, such as the probabilistic interpretation of linear regression, gaussian discriminantanalysis, mixture of Gaussians clustering, and most recently, factor analysis. In these lec-ture notes, we attempt to demystify some of the fancier properties of Multivariate Gaussiansthat were introduced in the recent factor analysis lecture. The goal of these notes is to giveyou some intuition into where these properties come from, so that you can use them withconfidence on your homework (hint hint!)

2 And DefinitionA vector-valued random variablex Rnis said to have amultivariate normal (or Gaus-sian) distributionwith mean Rnand covariance matrix Sn++1if its probabilitydensity function is given byp(x; , ) =1(2 )n/2| |1/2exp( 12(x )T 1(x )).We write this asx N( , ).2 gaussian factsMultivariate Gaussians turn out to be extremely handy in practice due to the following facts: Fact #1:If you know the mean and covariance matrix of a gaussian randomvariablex, you can write down the probability density function from the section notes on linear algebra thatSn++is the space of symmetric positive definiten nmatrices, defined asSn++={A Rn n:A=ATandxTAx >0 for allx Rnsuch thatx6= 0}.

3 1 Fact #2:The following gaussian integrals have closed-form solutions: x Rnp(x; , )dx= p(x; , )dx1.. dxn= 1 x Rnxip(x; , 2)dx= i x Rn(xi i)(xj j)p(x; , 2)dx= ij. Fact #3: Gaussians obey a number ofclosureproperties: The sum of independent gaussian random variables is gaussian . The marginal of a joint gaussian distribution is gaussian . The conditional of a joint gaussian distribution is first glance, some of these facts, in particular facts #1 and #2,may seem eitherintuitively obvious or at least plausible. What is probably not so clear, however, is whythese facts are so powerful.

4 In this document, we ll provide some intuition for how these factscan be used when performing day-to-day manipulations dealing with Multivariate Gaussianrandom Closure propertiesIn this section, we ll go through each of the closure properties described earlier, and we lleither prove the property using facts #1 and #2, or we ll at least give some type of intuitionas to why the property is following is a quick roadmap of what we ll cover:sums marginals conditionalswhy is it gaussian ?noyesyesresulting density Sum of independent Gaussians is GaussianThe formal statement of this rule is:Suppose thaty N( , ) andz N( , ) are independent gaussian dis-tributed random variables, where , Rnand , Sn++.

5 Then, their sumis also gaussian :y+z N( + , + ).Before we prove anything, here are some observations:21. The first thing to point out is that the importance of the independence assumption inthe above rule. To see why this matters, suppose thaty N( , ) for some meanvector and covariance matrix , and suppose thatz= y. Clearly,zalso has aGaussian distribution (in fact,z N( , ), buty+zis identically zero!2. The second thing to point out is a point of confusion for many students: if we addtogether two gaussian densities ( bumps in multidimensional space), wouldn t we getback some bimodal ( , two-humped density)?)

6 Here, the thing to realize is that thedensity of the random variabley+zin this rule is NOT found by simply adding thedensities of the individual random variablesyandz. Rather, the density ofy+zwillactually turn out to be aconvolutionof the densities show that theconvolution of two gaussian densities gives a gaussian density, however, is beyond thescope of this , let s just use the observation that the convolution does give some type of Gaus-sian density, along with Fact #1, to figure out what the density,p(y+z| , ) would be, ifwe were to actually compute the convolution.

7 How can we do this?Recall that from Fact#1, a gaussian distribution is fully specified by its mean vector and covariance matrix. Ifwe can determine what these are, then we re this is easy! For the mean, we haveE[yi+zi] =E[yi] +E[zi] = i+ ifrom linearity of expectations. Therefore, the mean ofy+zis simply + . Also, the(i, j)th entry of the covariance matrix is given byE[(yi+zi)(yj+zj)] E[yi+zi]E[yj+zj]=E[yiyj+ziyj+yizj+zizj] (E[yi] +E[zi])(E[yj] +E[zj])=E[yiyj] +E[ziyj] +E[yizj] +E[zizj] E[yi]E[yj] E[zi]E[yj] E[yi]E[zj] E[zi][zj]= (E[yiyj] E[yi]E[yj]) + (E[zizj] E[zi]E[zj])+ (E[ziyj] E[zi]E[yj]) + (E[yizj] E[yi]E[zj]).

8 Using the fact thatyandzare independent, we haveE[ziyj] =E[zi]E[yj] andE[yizj] =E[yi]E[zj]. Therefore, the last two terms drop out, and we are left with,E[(yi+zi)(yj+zj)] E[yi+zi]E[yj+zj]= (E[yiyj] E[yi]E[yj]) + (E[zizj] E[zi]E[zj])= ij+ example, ifyandzwere univariate Gaussians ( ,y N( , 2),z N( , 2)), then theconvolution of their probability densities is given byp(y+z; , , 2, 2) = p(w; , 2)p(y+z w; , 2)dw= 1 2 exp( 12 2(w )2) 1 2 exp( 12 2(y+z w )2)dw3 From this, we can conclude that the covariance matrix ofy+zis simply +.

9 At this point, take a step back and think about what we have just done. Using somesimple properties of expectations and independence, we havecomputed the mean and co-variance matrix ofy+z. Because of Fact #1, we can thus write down the density fory+zimmediately, without the need to perform a convolution! Marginal of a joint gaussian is GaussianThe formal statement of this rule is:Suppose that[xAxB] N([ A B],[ AA AB BA BB]),wherexA Rm,xB Rn, and the dimensions of the mean vectors and covariancematrix subblocks are chosen to matchxAandxB.

10 Then, the marginal densities,p(xA) = xB Rnp(xA, xB; , )dxBp(xB) = xA Rmp(xA, xB; , )dxAare gaussian :xA N( A, AA)xB N( B, BB).To justify this rule, let s just focus on the marginal distribution with respect to the , note that computing the mean and covariance matrix fora marginal distributionis easy: simply take the corresponding subblocks from the mean and covariance matrix ofthe joint density. To make sure this is absolutely clear, let s look at the covariance betweenxA,iandxA,j(theith component ofxAand thejth component ofxA). Note thatxA,iandxA,jare also theith andjth components of[xAxB]3Of course, we needed to know thaty+zhad a gaussian distribution in the first general, for a random vectorxwhich has a gaussian distribution, we can always permute entries ofxso long as we permute the entries of the mean vector and the rows/columns of the covariance matrix inthe corresponding way.