Probability Theory Review for Machine Learning

1 Probability Theory Review for Machine LearningSamuel IeongNovember 6, 20061 Basic ConceptsBroadly speaking, Probability Theory is the mathematical study of uncertainty. It plays acentral role in Machine Learning , as the design of Learning algorithms often relies on proba-bilistic assumption of the data. This set of notes attempts to cover some basic probabilitytheory that serves as a background for the Probability SpaceWhen we speak about Probability , we often refer to the Probability of aneventof uncertainnature taking place.

2 For example, we speak about the Probability of rain next , in order to discuss Probability Theory formally, we must first clarify what thepossible events are to which we would like to attach , aprobability spaceis defined by the triple ( ,F, P), where is thespace of possible outcomes(oroutcome space), F 2 (the power set of ) is thespace of (measurable) events(orevent space), Pis theprobability measure(orprobability distribution) that maps an eventE Ftoa real value between 0 and 1 (think ofPas a function).Given the outcome space , there is some restrictions as to what subset of 2 can beconsidered an event spaceF: The trivial event and the empty event is inF.

3 The event spaceFis closed under (countable) union, , if , F, then F. The even spaceFis closed under complement, , if F, then ( \ ) we throw a (six-sided) dice. The space of possible outcomes ={1,2,3,4,5,6}. We may decide that the events of interest is whether the dice throw is oddor even. This event space will be given byF={ ,{1,3,5},{2,4,6}, }.1 Note that when the outcome space is finite, as in the previous example, we often takethe event spaceFto be 2 . This treatment is not fully general, but it is often sufficientfor practical purposes.

4 However, when the outcome space is infinite, we must be careful todefine what the event space an event spaceF, the Probability measurePmust satisfy certain axioms. (non-negativity) For all F,P( ) 0. (trivial event)P( ) = 1. (additivity) For all , Fand = ,P( ) =P( ) +P( ).Example to our dice example, suppose we now take the event spaceFto be2 . Further, we define a Probability distributionPoverFsuch thatP({1}) =P({2}) = =P({6}) = 1/6then this distributionPcompletely specifies the Probability of any given event happening(through the additivity axiom).

5 For example, the Probability of an even dice throw will beP({2,4,6}) =P({2}) +P({4}) +P({6}) = 1/6 + 1/6 + 1/6 = 1/2since each of these events are Random VariablesRandom variablesplay an important role in Probability Theory . The most important factabout random variables is that they arenotvariables. They are actuallyfunctionsthatmap outcomes (in the outcome space) to real values. In terms of notation, we usually denoterandom variables by a capital letter. Let s see an , consider the process of throwing a dice. LetXbe a random variable thatdepends on the outcome of the throw.

6 A natural choice forXwould be to map the outcomeito the valuei, , mapping the event of throwing an one to the value of1. Note thatwe could have chosen some strange mappings too. For example, we could have a randomvariableYthat maps all outcomes to0, which would be a very boring function, or a randomvariableZthat maps the outcomeito the value of2iifiis odd and the value of iifiiseven, which would be quite strange a sense, random variables allow us to abstract away from the formal notion of eventspace, as we can define random variables that capture the appropriate events.

7 For example,consider the event space of odd or even dice throw in Example 1. We could have defined arandom variable that takes on value 1 if outcomeiis odd and 0 otherwise. These type ofbinary random variables are very common in practice, and are known asindicator variables,taking its name from its use to indicate whether a certain event has happened. So whydid we introduce event space? That is because when one studies Probability Theory (more2rigorously) using measure Theory , the distinction between outcome space and event spacewill be very important.

8 This topic is too advanced to be covered in this short Review any case, it is good to keep in mind that event space is not always simply the power setof the outcome here onwards, we will talk mostly about Probability with respect to random vari-ables. While some Probability concepts can be defined meaningfully without using them,random variables allow us to provide a more uniform treatment of Probability Theory . Fornotations, the Probability of a random variableXtaking on the value ofawill be denotedby eitherP(X=a) orPX(a)We will also denote the range of a random variableXbyV al(X).

9 Distributions, Joint Distributions, and Marginal DistributionsWe often speak about thedistributionof a variable. This formally refers to the probabilityof a random variable taking on certain values. For example,Example random variableXbe defined on the outcome space of a dice throw(again!). If the dice is fair, then the distribution ofXwould bePX(1) =PX(2) = =PX(6) = 1/6 Note that while this example resembles that of Example 2, they have different semanticmeaning. The Probability distribution defined in Example 2 is overevents, whereas the onehere is defined overrandom notation, we will useP(X) to denote the distribution of the random , we speak about the distribution of more than one variables at a time.

10 Wecall these distributionsjoint distributions, as the Probability is determined jointly by all thevariables involved. This is best clarified by an a random variable defined on the outcome space of a dice throw. LetYbe an indicator variable that takes on value1if a coin flip turns up head and0if both the dice and the coin are fair, the joint distribution ofXandYis given byPX= 1X= 2X= 3X= 4X= 5X= 6Y= 01/121/121/121/121/121/12Y= 11/121/121/121/121/121/12As before, we will denote the Probability ofXtaking valueaandYtaking valuebbyeither the long hand ofP(X=a, Y=b), or the short hand ofPX,Y(a, b).