Lecture Notes on Bayesian Estimation and …

1 This is page 1 Printer: Opaque thisLecture Notes on Bayesian Estimation andClassificationM ario A. T. Figueiredo,Instituto de Telecomunica c oes, andInstituto Superior T ecnico1049-001 LisboaPortugal(Email address: Copyright by the author. DO NOT 20042 This is page 3 Printer: Opaque thisContents1 Introduction to Bayesian Decision Introduction .. Statistical Decision Theory .. Basic Elements .. Frequentist Risk Function and Decision Rules .. Bayesian Decision Theory .. Subjective Probabilities and Degrees of Belief .. A Posteriori Expected Loss and Bayesian Decisions . Bayes Risk .. Admissibility of Bayesian Rules.)

2 Predictive Problems .. Inference versus Decision .. Bayesian Classification .. Introduction .. Classification Under The 0/1 Loss Function .. A Special Case: Gaussian Observations .. General Costs .. Discriminant Functions and Decision Regions .. Error Probabilities .. Bayesian Estimation .. Introduction .. The 0/1 loss function .. The quadratic error loss function .. The absolute error loss function .. 494 Summary .. 512 Topics in Bayesian Specifying Priors .. Improper Priors and Maximum Likelihood Estimation .. Conjugate priors .. Mixtures of Conjugate Priors.

3 Asymptotic Behavior Of Bayes Rules .. Non-informative Priors and Invariance .. Discrete Problems .. Location Parameters .. Scale Parameters .. Jeffreys Priors and Fisher Information .. Fisher Information .. Fisher Information in Asymptotic Normality .. Jeffreys Priors .. Maximum Entropy Priors .. Maximum Entropy Priors for Discrete Problems .. The Kullback-Leibler Divergence .. Maximum Entropy Priors for Estimation Problems . Mutual Information, Data ProcessingInequality, and Jeffreys Priors .. Conditional Entropy and Mutual Information .. The Data Processing Inequality .. An Information Theoretic View of Jeffreys Priors.

4 Minimum Description Length Priors .. Sufficient Statistics .. The Sufficiency Principle .. An Information Theoretic Perspective .. Exponential Families .. Fundamental Concepts .. Partition Function, Free Energy, and Entropy .. Conjugate Priors .. Fisher Information and Jeffreys Priors .. Intrinsic Loss Functions and Density Estimation Problems . Foundational Aspects: The Likelihood, Conditionality, andSufficiency Principles .. Summary .. 1193 Compound Decision Theory and Contextual Decision Introduction .. Non-additive Loss Functions .. The 0/1 Loss Function .. Quadratic Loss Function .. 124 Contents Likelihood Related Loss Functions.

5 Linear-Gaussian Observations and Gaussian Priors .. Regression Models .. Gaussian Prior: General Solution .. Particular Cases .. Additive Loss Functions .. General Result: Marginal Criteria .. Application to the common loss functions .. Priors for Compound Problems .. Improper Priors and Maximum Likelihood Inference Exponential Families and Conjugate Priors .. Non-informative Jeffreys Priors .. Maximum Entropy Priors .. Independence Maximizes Entropy .. Discrete Problems .. Estimation Problems .. Maximum Entropy and Exponential Families .. Summary .. 155A Notation157B Markov Processes and Chains: a Brief Discrete-Index Stochastic Processes.

6 Markov Processes .. Irreducibility and Stationary Distributions .. Chapman-Kolmogorov Equations .. Other Properties of Markov Processes .. 162 References1656 ContentsThis is page 7 Printer: Opaque this1 Introduction to Bayesian IntroductionStatistical decision theory deals with situations where decisions have to bemade under a state of uncertainty, and its goal is to provide a rationalframework for dealing with such situations. The Bayesian approach, themain theme of this chapter, is a particular way of formulating and dealingwith statistical decision problems. More specifically, it offers a method offormalizinga prioribeliefs and of combining them with the available obser-vations, with the goal of allowing a rational (formal) derivation of optimal(in some sense) decision can be inferred from the previous paragraph, this book s introductionto Bayesian theory adopts a decision theoretic perspective.

7 An importantreason behind this choice is that inference problems ( , how to estimatean unknown quantity) can be naturally viewed as special cases of decisionproblems; this way, all the conceptual tools of Bayesian decision theory(a prioriinformation and loss functions) are incorporated into literature on Bayesian theory is vast and anyone interested in fur-ther reading is referred to the many excellent textbooks available on thesubject; at the risk of unfairly but unintentionally leaving out importantworks, we mention here some books that somehow influenced the authors:Berger [8], Bernardo and Smith [14], Gelman, Carlin, Stern, and Rubin[46], Lee [69], and Robert [93]; a not recent, but still useful and insightfulreview is the one by Lindley [72]; a short and easily readable summary of81.

8 Introduction to Bayesian Decision Theorythe main arguments in favor of the Bayesian perspective can be found ina paper by Berger whose title, Bayesian Salesmanship, clearly revealsthe nature of its contents [9]. Also highly recommended by its conceptualdepth and the breadth of its coverage is Jaynes (still unfinished but par-tially available) book [58]. Recent advances are reported in workshops andconferences (special emphasis should be be given to [13], [11], and [12])and in several scientific journals (for example, theJournal of the AmericanStatistical Associationand theJournal of the Royal Statistical Society). Bayesian frameworks have been used to deal with a wide variety of prob-lems in many scientific and engineering areas.

9 Whenever a quantity is to beinferred, or some conclusion is to be drawn, from observed data, Bayesianprinciples and tools can be used. Examples, and this is by no means anexhaustive list of mutually exclusive areas, include: statistics, signal pro-cessing, speech analysis, image processing, computer vision, astronomy,telecommunications, neural networks, pattern recognition, machine learn-ing, artificial intelligence, psychology, sociology, medical decision making,econometrics, and biostatistics. Focusing more closely on the topic of inter-est to this book, we mention that, in addition to playing a major role inthedesign of machine (computer) vision techniques, the Bayesian frameworkhas also been found very useful in understanding natural ( , human)perception [66].

10 This fact is a strong testimony in favor of the , it is worth pointing out that the Bayesian perspective is not onlyimportant at a practical application level, but also at deeper conceptual lev-els, touching foundational and philosophical aspects of scientific inference,as the title of Rozenkrantz s book [95] so clearly shows: Inference, Method,and Decision: Towards a Bayesian Philosophy of Science . On this issue,the book by Jaynes is a fundamental more recent reference [58]. Statistical Decision Basic ElementsThe fundamental conceptual elements supporting the (formal) theory ofstatistical decision making are the following: Formalization of the underlying unknown reality.