Probability Theory - Wharton Faculty

1 1 Contract no: BAP098M-RG-15-MedProbability SteeleWhartonProbability Theory is that part of mathematics that aims to provide insight into phe-nomena that depend on chance or on uncertainty. The most prevalent use of thetheory comes through the frequentists interpretation of Probability in terms of theoutcomes of repeated experiments, but Probability is also used to provide a measureof subjective beliefs, especially as judged by one s willingness to place roots of Probability Theory are not as ancient as those of many parts of math-ematics, and only in the sixteenth and seventeenth centuries does one find the firstglimmerings of the Theory in the investigations made by Gerolamo Cardano, Pierrede Fermat, and Blaise Pascal into games of chance.

2 Despite the luminous reputa-tions of these famous mathematicians and philosophers, the subject of probabilitytheory remained on the periphery of respectability, and for a long time developmentwas halting and lugubrious. Through the first third of the twentieth century, theeighteenth century works of Jakob Bernoulli (see Bernoulli Family) and AbrahamDe Moivre continued to be viewed as the nearly definitive treatises of Encyc. Social and Behavioral Sciences18 March 20032 Still, even in the early days of the twentieth century when Probability Theory clearlysuffered from the lack of a widely accepted foundation, there were profound devel-opments, most notably Albert Einstein s use of Brownian motion in 1905 to pro-vide the first determination of Avagadro s number [7].

3 Nevertheless, in 1933 whenAndrey Nikolayevich Kolmogorov published his elegant succinct volumeFounda-tions of Probability Theory [10], the mathematical world was hungry for such atreatment, and the subsequent development of Probability Theory was Firm FoundationCentral to Kolmogorov s foundation for Probability Theory was his introduction ofthe triple( ,F,P)that we now call a Probability space, or sometimes the proba-bilist s trinity . The triple s first element, , is required only to be a set. The secondelement is a collection of subsets of about which more will be said later. The thirdelement is a function that assigns a real number to each of the elements ofF.

4 Thisfunction is called a Probability measurePprovided that it satisfies the three fol-lowing axioms:Axiom 1. For allA Fwe haveP(A) 2. For any countable collection{Ai F: 1 i < }for whichAi Aj= for alli6=j, we haveP( i=1Ai) = i=1P(Ai).3 Axiom ( ) = 1 Axioms 1 and 3 are quite bland. Axiom 1 only captures ourunderstanding that probabilities of events are nonnegative numbers, and Axiom 3just echoes our assumption that is a sensible representation for the universe of allpossible outcomes of the chance experiment being modeled. Only about Axiom 2can there can be any quarrel, and at times arguments have been made for preferringa Probability Theory that only requires additivity of probabilities for finite collec-tions of sets.

5 Kolmogorov s decision to assume countable additivity is not the onlypossible choice, but it has been a fecund one that has proved to be appropriate in awide variety of mathematical benefit of Kolmogorov s second axiom is that it connects proba-bility Theory with the Theory of measure as put forward by Borel, Lebesgue, Radon,and Fr echet in the early part of the twentieth century. It was in fact Fr echet whonoted some 13 years after Lebesgue s famous 1902 thesis that the natural domainfor a Probability measure is a collection of sets that is closed under complemen-tation and countable unions. Fr echet called such collections -algebras, and Kol-mogorov required that the second term of his triple be just such a Basic Quantities of the TheoryTo the practical mind, Kolmogorov s axiomatization of Probability may seem onlyto defer the problem of construction of Probability models that serve to inform usabout the physical and social world, but by putting the elusive Probability functionPon an axiomatic footing Kolmogorov did provide real assurance that one could4study Probability as sensibly as one could study measure Theory , analysis or particular.

6 One could proceed with the investigation of the objects that had beenof concern from Probability s earliest of the most fundamental notions of Probability Theory is the random variable,and in Kolmogorov s framework a random variable is nothing more than a functionfromX: <with the property that for alltone has that the sets{ :X( ) t}are elements of the -algebraF. With this definition we are on firm footing whenwe take the definition of the distribution functionFofXto beF(t) =P(X t),because the set{ :X( ) t}is in the domain of the set functionP. In this frame-work the expectationE(X)of the random variableXcan defined as the Lebesgueintegral ofXwith regard toP, or as the Riemann-Stieltjes integral with respect toF, giving usE(X) X( )dP( ) = xdF(x).

7 The Probability distribution function and the expectation operation provide us withthe core language that is needed to express almost everything that one needs to sayabout individual random variables. For example, a basic measure of dispersion of arandom variable is the variance, which one writes in terms of the expectation as5var(X) E(X )2,where =E(X)and the standard deviation ofXis defined to be the square rootof the Central Role of IndependenceWith expectations and distributions we recapture much of the most basic languageof Probability Theory , but the real power of Probability Theory only emerges with theintroduction of the central notion ofindependenceof events, algebras, and randomvariables.

8 To begin that development, one first defines elementsAandBofFtobe independent providedP(A B) =P(A)P(B).This definition is then extended to sub- -algebras ofAandBofFby callingAandBindependent providedAandBare independent for allA Aand allB , random variablesXandYare independent ifAandBindependent whenthese are respectively the smallest -algebras containing all the sets{X t}andall the sets{Y t}.This definition of independence of random variables may look a little burdensomeat first, but for many purposes it is much more convenient than the definition ofindependence that is sometimes given in elementary texts that call for the factor-6ization of the joint density ofXandY.

9 In fact densities may not exist, but that isnot the telling point. More to the heart of the matter is that with Kolmogorov s defi-nition one clearly sees that the independence ofXandYimplies the independenceoff(X)andg(Y)for any monotone functionsfandg, while this intuitive fact iscumbersome to check if one needs to verify a density Theorems That Make the TheoryThere are two theorems that live at the very heart of Probability Theory . The firstis the law of large numbers, without which our most fundamental intuitions aboutthe relationship of Probability Theory and the physical world would be at odds. Thesecond is the central limit theorem, which is arguably the result that most clearlyaccounts for the practical utility of Probability as a helpmate to statistics, as well asto the social and physical 1 (Law of Large Numbers).

10 If{Xi: 1 i < }is a sequence ofindependent random variables, with the distribution function,F, and ifE|Xi|< , then the event that the sequence1n{X1+X2+..+Xn}converges toE(X1)has Probability 2 (Central Limit Theorem). If{Xi: 1 i < }is a sequence ofindependent random variables with distribution functionF, E(Xi) = < , and7var(X) = 2< , thenlimn P(1 n{X1+X2+..+Xn n } x)=1(2 )1/2 x e 2 Beyond Independent Random VariablesWhile the purest view of the aims and accomplishments of Probability Theory maybe found in the study of sums of independent random variables, the applications ofprobability Theory require the development of structures that also capture aspectsof dependence.