Probability Theory: The Logic of Science

1 Probability Theory: The Logic of SciencebyE. T. JaynesWayman Crow Professor of PhysicsWashington UniversitySt. Louis, MO 63130, U. S. to the Memory of Sir Harold Jeffreys,who saw the truth and preserved 1995 by Edwin T. FORWARDE. T. Jaynes died April 30, 1998. Before his death he asked me to finish and publish his bookon Probability theory. I struggled with this for some time, because there is no doubt in my mindthat Jaynes wanted this book finished. Unfortunately, most of the later Chapters, Jaynes intendedvolume 2 on applications, were either missing or incomplete and some of the early also Chaptershad missing pieces.

2 I could have written these latter Chapters and filled the missing pieces, but if Idid so, the work would no longer belong to Jaynes; rather, it would be a Jaynes-Bretthorst hybridwith no way to tell which material came from which author. In the end, I decided that the missingChapters would have to stay missing the work would remain Jaynes .There were a number of missing pieces of varying length that Jaynes had marked by insertingthe phrase MUCH MORE COMING. I could have left these comments in the text, but theywere ugly and they made the book looks very incomplete.

3 Jaynes intended this book to serve asboth a reference and a text book. Consequently, there are question boxes scattered throughoutmost Chapters. In the end, I decided to replace the MUCH MORE COMING comments byintroducing an editors question box. If you answer these questions, you will have filled in themissing material. You will be able to identify these questions because I used a shaded box for theeditors questions, while Jaynes question boxes are not wanted to include a series of computer programs that implemented some of the calcu-lations in this book.

4 I had originally intended to include these programs. But as time went on, itbecame increasingly obvious that many of the programs were not available and the ones that were,were written in a particularly obscure form of BASIC (it was the programs that were obscure, notthe BASIC). Consequently, I removed references to these programs and, where necessary, inserteda few sentences to direct people to the necessary software tools to implement the , while I am the most obvious person who has worked on getting this book into pub-lication, I am not the only person to do so.

5 Some of Jaynes closest friends have assisted me incompleting this work. These include Tom Grandy, Ray Smith, Tom Loredo, Myron Tribus andJohn Skilling, and I would like to thank them for their assistance. I would also like to thank JoeAckerman for allowing me to take the time necessary to get this work Larry Bretthorst, EditorMay 2002iiPROBABILITY THEORY THE Logic OF SCIENCEVOLUME I PRINCIPLES AND ELEMENTARY APPLICATIONSC hapter 1 Plausible Reasoning1 Deductive and Plausible Reasoning1 Analogies with Physical Theories3 The Thinking Computer4 Introducing the Robot5 Boolean Algebra6 Adequate Sets of Operations9 The Basic Desiderata12 Comments15 Common Language vs.

6 Formal Logic16 Nitpicking18 Chapter 2 The Quantitative Rules21 The Product Rule21 The Sum Rule26 Qualitative Properties31 Numerical Values32 Notation and Finite Sets Policy38 Comments39 Subjective vs. Objective 39G odel s Theorem39 Venn Diagrams42 The Kolmogorov Axioms 43 Chapter 3 Elementary sampling Theory45 sampling Without Replacement45 Logic Versus Propensity52 Reasoning from Less Precise Information56 Expectations58 Other Forms and Extensions59 Probability as a Mathematical Tool60 The Binomial Distribution61 sampling With Replacement63 Digression: A Sermon on Reality vs.

7 Models64 Correction for Correlations66 Simplification71 Comments72A Look Ahead74 Chapter 4 Elementary Hypothesis Testing77 Prior Probabilities77 Testing Binary Hypotheses with Binary Data80 Non-Extensibility Beyond the Binary Case86 Multiple Hypothesis Testing88iiiContinuous Probability Distribution Functions (pdf s)95 Testing an Infinite Number of Hypotheses97 Simple and Compound (or Composite) Hypotheses102 Comments103 Etymology103 What Have We Accomplished?104 Chapter 5 Queer Uses For Probability Theory107 Extrasensory Perception107 Mrs. Stewart s Telepathic Powers107 Digression on the Normal Approximation109 Back to Mrs.

8 Stewart109 Converging and Diverging Views113 Visual Perception Evolution into Bayesianity?118 The Discovery of Neptune119 Digression on Alternative Hypotheses121 Back to Newton123 Horse racing and Weather Forecasting124 Discussion127 Paradoxes of Intuition128 Bayesian Jurisprudence128 Comments130 Chapter 6 Elementary Parameter Estimation133 Inversion of the Urn Distributions133 BothNandRUnknown133 Uniform Prior135 Predictive Distributions137 Truncated Uniform Priors139A Concave Prior141 The Binomial Monkey Prior143 Metamorphosis into Continuous Parameter Estimation145 Estimation with a Binomial sampling Distribution146 Digression on

9 Optional Stopping148 Compound Estimation Problems149A Simple Bayesian Estimate: Quantitative Prior Information150 From Posterior Distribution Function to Estimate153 Back to the Problem156 Effects of Qualitative Prior Information158 Choice of a Prior159On With the Calculation!160 The Jeffreys Prior162 The Point of It All164 Interval Estimation166 Calculation of Variance167 Generalization and Asymptotic Forms168 Rectangular sampling Distribution170 Small Samples172ivMathematical Trickery172 Comments174 Chapter 7 The Central, Gaussian Or Normal Distribution177 The Gravitating Phenomenon177 The Herschel-Maxwell Derivation178 The Gauss Derivation180 Historical Importance of Gauss Result180 The Landon Derivation182 Why the Ubiquitous Use of Gaussian Distributions?

10 185 Why the Ubiquitous Success?187 What Estimator Should We Use?188 Error Cancellation190 The Near-Irrelevance of sampling Frequency Distributions192 The Remarkable Efficiency of Information Transfer193 Other sampling Distributions194 Nuisance Parameters as Safety Devices195 More General Properties196 Convolution of Gaussians197 The Central Limit Theorem197 Accuracy of Computations200 Galton s Discovery202 Population Dynamics and Darwinian Evolution204 Evolution of Humming-Birds and Flowers206 Application to Economics207 The Great Inequality of Jupiter and Saturn208 Resolution of Distributions into Gaussians209