Chapter 12 Bayesian Inference - Carnegie Mellon University

1 Chapter 12 Bayesian InferenceThis Chapter covers the following topics: Concepts and methods of Bayesian Inference . Bayesian hypothesis testing and model comparison. Derivation of the Bayesian information criterion (BIC). Simulation methods and Markov chain Monte Carlo (MCMC). Bayesian computation via variational Inference . Some subtle issues related to Bayesian What is Bayesian Inference ?There are two main approaches to statistical machine learning:frequentist(or classical)methods andBayesianmethods. Most of the methods we have discussed so far are fre-quentist.

2 It is important to understand both approaches. At the risk of oversimplifying, thedifference is this:Frequentist versus Bayesian Methods In frequentist Inference , probabilities are interpreted as long run goal is to create procedures with long run frequency guarantees. In Bayesian Inference , probabilities are interpreted as subjective degrees of be-lief. The goal is to state and analyze your Machine LearningCHAPTER INFERENCESome differences between the frequentist and Bayesian approaches are as follows:FrequentistBayesianProbability is:limiting relative frequency degree of beliefParameter is a:fixed constantrandom variableProbability statements are about:proceduresparametersFrequency guarantees?

3 YesnoTo illustrate the difference, consider the following example. Suppose thatX1,..,Xn N( ,1). We want to provide some sort of interval estimateCfor .Frequentist the confidence intervalC= Xn ,Xn+ .ThenP ( 2C)= all probability statement is about the random intervalC. The interval is random becauseit is a function of the data. The parameter is a fixed, unknown quantity. The statementmeans thatCwill trap the true value with make the meaning clearer, suppose we repeat this experiment many times. In fact, wecan even allow to change every time we do the experiment.

4 The experiment looks likethis:Naturechooses 1 !Nature generatesndata points fromN( 1,1) !Statistician computesconfidence intervalC1 Naturechooses 2 !Nature generatesndata points fromN( 2,1) !Statistician computesconfidence will find that the intervalCjtraps the parameter j, 95 percent of the time. Moreprecisely,lim infn!11nnXi=1I( i2Ci) ( )almost surely, for any sequence 1, 2,.. Bayesian Bayesian treats probability as beliefs, not frequencies. Theunknown parameter is given a prior distributon ( )representing his subjective beliefs300 Statistical Machine Learning, by Han Liu and Larry Wasserman, c 2014 Statistical Machine WHAT IS Bayesian Inference ?

5 About . After seeing the dataX1,..,Xn, he computes the posterior distribution for given the data using Bayes theorem: ( |X1,..,Xn)/L( ) ( )( )whereL( )is the likelihood function. Next we finds an intervalCsuch thatZC ( |X1,..,Xn)d = can thn report thatP( 2C|X1,..,Xn)= is a degree-of-belief probablity statement about given the data. It is not the sameas ( ). If we repeated this experient many times, the intervals wouldnottrap the truevalue 95 percent of the Inference is aimed at given procedures with frequency guarantees.

6 Bayesianinference is about stating and manipulating subjective beliefs. In general, these are differ-ent, A lot of confusion would be avoided if we usedF(C)to denote frequency probablityandB(C)to denote degree-of-belief probability. These are idfferent things and there isno reason to expect them to be the same. Unfortunately, it is traditional to use the samesymbol, such asP, to denote both types of probability which leads to summarize: Frequentist Inference gives procedures with frequency probability guar-antees.

7 Bayesian Inference is a method for stating and updating beliefs. A frequentistconfidence intervalCsatisfiesinf P ( 2C)=1 where the probability refers to random intervalC. We callinf P ( 2C)the coverage ofthe intervalC. A Bayesian confidence intervalCsatisfiesP( 2C|X1,..,Xn)=1 where the probability refers to . Later, we will give concrete examples where the coverageand the posterior probability are very are, in fact, many flavors of Bayesian Inference . Subjective Bayesians in-terpret probability strictly as personal degrees of belief.

8 Objective Bayesians try to findprior distributions that formally express ignorance with the hope that the resulting poste-rior is, in some sense, objective. Empirical Bayesians estimate the prior distribution fromthe data. Frequentist Bayesians are those who use Bayesian methods only when the re-sulting posterior has good frequency behavior. Thus, the distinction between Bayesian andfrequentist Inference can be somewhat murky. This has led to much confusion in statistics,machine learning and Machine Learning, by Han Liu and Larry Wasserman, c 2014301 Statistical Machine LearningCHAPTER Basic ConceptsLetX1.

9 ,Xnbenobservations sampled from a probability densityp(x| ). In this Chapter ,we writep(x| )if we view as a random variable andp(x| )represents the conditionalprobability density ofXconditioned on . In contrast, we writep (x)if we view as adeterministic The Mechanics of Bayesian InferenceBayesian Inference is usually carried out in the following choose a probability density ( ) called the prior distribution thatexpresses our beliefs about a parameter before we see any choose a statistical modelp(x| )that reflects our beliefs aboutxgiven.

10 Observing dataDn={X1,..,Xn}, we update our beliefs and calculatethe posterior distributionp( |Dn).By Bayes theorem, the posterior distribution can be written asp( |X1,..,Xn)=p(X1,..,Xn| ) ( )p(X1,..,Xn)=Ln( ) ( )cn/Ln( ) ( )( )whereLn( )=Qni=1p(Xi| )is the likelihood function andcn=p(X1,..,Xn)=Zp(X1,..,Xn| ) ( )d =ZLn( ) ( )d is the normalizing constant, which is also called the can get a Bayesian point estimate by summarizing the center of the posterior. Typically,we use the mean or mode of the posterior distribution.