1 Multivariate Normal Distribution - Princeton University

1 STA561: Probabilistic machine learningGaussian Models (9/9/13)Lecturer: Barbara EngelhardtScribes: Xi He, Jiangwei Pan, Ali Razeen, Animesh Srivastava1 Multivariate Normal DistributionThe Multivariate Normal Distribution (MVN), also known asmultivariate gaussian, is a generalization ofthe one-dimensional Normal Distribution to higher dimensions. The probability density function (pdf) of anMVN for a random vectorx Rdas follows:N(x| , ),1(2 )d/2| |1/2exp[ 12(x )T 1(x )](1)where =E[x] Rdis the mean vector, and =cov[x] isd dsymmetric positive definite matrix, knownas the covariance matrix. 1is known as the precision 1 21steigenvector of 2ndeigenvector of Figure 1: 2 dimensional Gaussian 1 shows a 2-dimensional Gaussian density. The random vectors span two dimensions and are denoted inthe plot byX1(x-axis) andX2(y-axis). The means ofX1andX2are 1and 2respectively.

2 The density at is highest, and as the random vector moves away from , the density goes down. All of the points on thered contour (level set) have the same density. The first and second eigenvectors of the covariance matrix areorthogonal to each other as shown in the Fig. 1. The first eigenvalue is the direction of maximum variancein the MVN; the second eigenvector is orthogonal to the expression inside the exponent can be rewritten as: (x )T 1(x ). The Mahalanobis distancebetween two vectorsx1andx2is equivalent to the MVN, calculating the pdf of one of the two points, with12 Gaussian Modelsthe other as the mean:md(x1,x2) = (x1 x2)T 1(x1 x2).(2)Note that this distance is MLE for MVNIn order to determine the MLE for a MVN, we need some basic results from linear algebra. Recall thefollowing definitions. Thetraceof a matrixA Rd dis defined as the sum ofA s diagonal elements, ,tr(A) = diAii.

3 Thedeterminantof a matrixA Rd dis defined as the product of its eigenvalues. A positive definitematrixAhas positive eigenvalues, so the determinant will always be positive. Symmetric positive definite matrices (as we will consider for our covariance matrices) are defined ashaving eigenvalues that are strictly positive. Trace has thecyclic permutationproperty:tr(ABC) =tr(CAB) =tr(BCA)Given vectorsa,band matricesA,B,C, we have the following facts: bTa a=b (aTAa) a= (A+AT)a. (Note that ifAis symmetric, this equals to 2Aa) tr(BA) A=BT log|A| A=A T,(A 1)T Trace trick:aTAa=tr(aTAa) =tr(aaTA) =tr(AaaT)We will be using these aforementioned facts in deriving the MLEs, MLEand MLE, for a MVN given adata setD={x1,x2,x3,..,xn}, wherexi Rdis a sample vector from the MVN. The log likelihood of thedata setDgiven MVN parameters , can be written asL( , ;D) = logn i=1p(xi| , )(3)=n2log| 1| 12n i=1tr[(xi )(xi )T 1](4)Set the partial derivative with respect to to 0, L( , ;D) = 12n i=1 2 1(xi )] = 0 Gaussian Models3we get MLE of as follows,n i=1(xi ) = 0 n i=1xi n i=1 = 0 MLE=1nn i=1xi(5)This means that the MLE of for MVN is just the empirical mean of the , setting the partial derivative of the log likelihood (Equation 4) with respect to 1to 0, L( , ;D) 1=n2 12(n i=1(xi )(xi )T) = 0we get MLE of as follows, MLE=1nn i=1(xi )(xi )T(6)This expression is just the empirical covariance of the data, centered on.

4 3 The MVN is in the Exponential FamilyWe have already seen that ifx N( , ) thenE[x] = andcov[x] = . These are also called themeanor themomentparameters of the Distribution . We can also express the MVN in exponential family form interms of thenatural parametersas = 1, = 1 .(7)Similarly, we can convert the natural parameters back to moment parameters as = 1, = 1 .(8)Note that the natural parameter covariance matrix is the precision matrix. Also note that the relationshipbetween the mean parameters and the natural parameters is an invertible relationship, so the MLE for thenatural parameters can be converted into the MLE for the mean parameters (and vice versa). This enablesus to work in the most mathematically convenient space, and convert afterwards between can rewrite the MVN density, in Eqn (1), in exponential family form using the natural parameters asfollows:P(x| , ) = (2 ) d/2| |1/2exp[ 12(xT x+ T 2xT )]= (2 ) d/2| |1/2exp[ Tx 12xT x 12 T ]= (2 ) d/2| |1/2exp[ Tx 12tr( xxT) 12 T ](9)Recall the exponential family form:P(X| ) =h(X) exp{ TT(X) A( )},(10)4 Gaussian Modelswhere in Eqn (10) denotes the natural parameter vector, and T(x) is the vector of sufficient statistics forthe can see then that Eqn (9) is in exponential family form, and we read out the sufficient statistics of MVN.

5 T(x) =[xxxT],and the natural parameters are =[ 12 ]and the log partition function isA( , ) =12 T =12 , the sufficient statistics for a MVN are the empirical mean and the empirical Marginals and Conditionals for an MVNLet s consider an example whered= 2. If it is simpler, letX= (x1,x2) wherex1andx2are scalar. However,if we considerx1andx2to be a split of the MVN data in dimensiond >2, where eachx1andx2is a vector,all of this subsequent section goes through naturally. Suppose thatx1andx2are jointly Gaussian: =[ 1 2], =[ 11 12 21 22]and hence = 1=[ 11 12 21 22]Can we find the marginal and conditional distributions in this space? Recall thatP(x1) = x2N(x1,x2| , ) this model we can derive the following distributions with Eqn. (8): Marginal:P(x1) =N(x1| 1, 11), where 1= 2 21 111 1and 11= 22 21 111 12. Marginal (equivalent):P(x2) =N(x2| 2, 22), where 2= 1 12 122 2and 22= 11 12 122 21.

6 Conditional Distribution :P(x1|x2) =N(x1| 1|2, 1|2) , where 1|2= 1+ 12 122(x2 2), 1|2= 11 12 122 21, or more concisely (natural parameters), 1|2= 1 12x2, 1|2= converse conditional Distribution ,p(x2|x1) is written out equivalently (swapping the 1, 2 indices). Theseformulas are derived using the Schur complement of a matrix and the matrix inversion lemma. Note thatconditional probabilities are straightforward to consider in the natural parameter space, where marginalprobabilities are much simpler in the mean parameter marginal Distribution in the mean parameter space is a simple projection of a (for example) 2D MVNcloud onto each of the univariate Gaussian distributions i one dimension. The conditional Distribution is asimilar projection, but considering only a slice of the space at the conditional random variable. When theoff-diagonal elements of the covariance matrix are 0, the conditional Distribution is identical to the marginaldistribution (as the two univariate Gaussians are independent).

7 Gaussian Models55 Conjugate priorThe conjugate prior for the mean term of a Multivariate Normal Distribution is a Multivariate normaldistribution:p( |X) p( )p(X| ),(11)wherep( ) is a Multivariate Normal Distribution , N( 0, 0). The implication of this prior is that themean term has a Gaussian Distribution across the space that it might lie in: generally large values of 0are preferable unless we have good prior information about the mean term ( , that it will be right aroundzero).The conjugate prior for the covariance matrix of a Multivariate Normal Distribution is the inverse Wishartdistribution:p( |X) p( )p(X| ),(12)wherep( ) is an inverse Wishart Distribution IW( , ). The inverse Wishart is a PDF for positivedefinite matrices.