EE414 - Probability & Stochastic Processes

1 Probability & Stochastic ProcessesIntroduction to Probability TheorySample SpacesEvent SpacesProbability MeasureProbability FunctionsRandom VariablesMoments of Random VariablesIntroduction to Stochastic ProcessesDr Conor McArdleEE414 - Probability & Stochastic Processes1/60 Introduction to Probability TheoryProbability theory is concerned with the description and calculation of the properties ofrandom phenomena, as occur in games of chance, computer and telecommunicationssystems, financial markets, electronic and optical circuits and many other such systems are random, in the sense that it is difficult or impossible topredict exactly how the system will behave in the future, Probability theory can providecharacterisation of the type of randomness involved and yield useful measures, such asaverage values of system parameters or the likelihood of certain events occurring in develop a rigorous mathematical theory of Probability , the starting point is thenotion of arandom experimentand anabstract Probability experimentEis an experiment satisfying the following conditions:all possible distinct outcomes are known a priorithe outcome is not known a priori for any particular trial of the experimentthe experiment is repeatable under identical conditionsDr Conor McArdleEE414 - Probability & Stochastic Processes2/60 Introduction to Probability TheoryMany random phenomena can be modelled by the notion of a random experiment, forexample.

2 Recording the output voltage of a noise generatorObserving the daily closing price of crude oilMeasuring the number of packets queueing at the input port of a network routerEach different random experimentEdefines a its own particularsample space,eventspaceandprobability measure, which collectively form anabstract Probability spaceforthe random spaceis the collection ( ,F,P) where thesample spaceis the set of all possibleoutcomesof a randomexperimentEFtheevent spaceis a collection ofevents, where each event is a subset ofthe sample space and the collection forms a -fieldPtheprobability measureis an assignment of a real number in the interval[0,1] to each event in the event Conor McArdleEE414 - Probability & Stochastic Processes3/60 Introduction to Probability TheoryExample: Random experiment of tossing a fair coinSample Space ={H,T},Event SpaceF={{H},{T},{H,T},{}} Probability MeasurePdefined byP({H}) =12,P({T}) =12,P({H,T}) = 1,P({}) = 0 Considering this example where the sample space is discrete (countable), it may appearunnecessary to define events to which probabilities are assigned.

3 Why not simplyassign probabilities directly to outcomes in the sample space?Consider instead an experiment where a random selection of a real number between 0and 10 is made (anuncountablesample space), then the Probability of any particularoutcome must be zero since there is an infinity of such outcomes in the sample , ifeventsare defined as intervals of the real line ( [0,5]), the events canhave non-zero Probability values ( the Probability of an outcome occurring withinthe interval [0,5] will be non-zero).Dr Conor McArdleEE414 - Probability & Stochastic Processes4/60 Introduction to Probability TheorySo that we can form a useful theory for all random experiments (particularly those withuncountablesample spaces), the Probability measure is only defined on specifiedsubsetsof the sample space (the events) rather than on individual outcomes in thesample that this stipulation does not preclude us from defining events consisting of asingle outcome, but we draw the distinction between an outcome (an element of ) and an event{ } (a subset of ).

4 The definition of the event space as a -fieldfurther specifies which subsets of canbelong to the same event space. That is, there is a certain relationship between thesubsets of the sample space that are chosen as events in the event properties of a -field (and so of any event space) ensure that if eventsAandBhave probabilities defined then logical combinations of these events ( the outcomeis in eitherAorB) are also events in the event space and so also have probabilitiesdefined. Any subset of that does not belong to the event space of a randomexperiment will simply not have a defined next look at the sample space, event space and Probability measure in some Conor McArdleEE414 - Probability & Stochastic Processes5/60 Probability & Stochastic ProcessesIntroduction to Probability TheorySample SpacesEvent SpacesProbability MeasureProbability FunctionsRandom VariablesMoments of Random VariablesIntroduction to Stochastic ProcessesDr Conor McArdleEE414 - Probability & Stochastic Processes6/60 Sample SpacesAsample space is the non-empty set of all outcomes (also known assamplepoints,elementary outcomesorelementary events) of a random sample space takes different forms depending on the random experiment inquestion.

5 We have seen an example of a finite sample space{H,T}, in the case of thecoin tossing random experiment, and also an uncountable sample space (a interval ofthe real line[0,10]) in the case of the random number follows are some examples of more general sample spaces:Example 1A finite sample space ={ak:k= 1,2,..,K}. Specific examples are:A binary space{0,1}A finite space of integers{0,1,2,..,k 1}. (Also denotedZk).Dr Conor McArdleEE414 - Probability & Stochastic Processes7/60 Sample SpacesExample 2A countably infinite space ={ak:k= 1,2,..}. Specific examples are:All non-negative integers{0,1,2,..}, denotedZ+All integers{.., 2, 1,0,1,2,..}, denotedZExample 3An uncountably infinite space. Examples are the real lineRor intervals ofRsuch as(a,b),[a,b),(a,b],[a, ),( , ).Example 4A space consisting ofk-dimensional vectors with coordinates taking values in one ofthe previously described spaces. The usual name for such a vector space is aproductspace.]

6 For example, letAdenote one of the abstract spaces previously the cartesian productAkas:Ak={(ao,a1,..,ak 1) :ai A}Dr Conor McArdleEE414 - Probability & Stochastic Processes8/60 Sample SpacesSpecific examples of this type of space are:Rk{0,1}k[a,b]kExample 5 LetAbe one of the sample spaces in examples 1-3. Form a new sample spaceconsisting of all waveforms (or functions of time) with values inA( all real valuedtime functions). This space is a product space of infinite dimension. For example:At={all waveforms{x(t) :t [0, )}:x(t) A, t}Exercise 1 Specify appropriate sample spaces that model the outcomes of the following ran-dom systems: (i) tossing a coin where a head is assigned a value of 1 and a taila value of 0 (ii) rolling a die (iii) rolling three dice simultaneously (iv) choosing arandom coordinate within a cube (v) an infinite random binary Conor McArdleEE414 - Probability & Stochastic Processes9/60 Probability & Stochastic ProcessesIntroduction to Probability TheorySample SpacesEvent SpacesProbability MeasureProbability FunctionsRandom VariablesMoments of Random VariablesIntroduction to Stochastic ProcessesDr Conor McArdleEE414 - Probability & Stochastic Processes10/60 Event SpacesTheevent spaceFof a sample space is a non-empty collection of subsets of , which has the following properties:1 IfF Fthen alsoFc F2If for some finiten,Fi F, i= 1,2.]

7 ,nthen alson i=1Fi F3 IfFi F, i= 1,2,..then also i=1Fi FThese properties specify that an event space is a -field(or -algebra) over .Note that the definition of the -field, as above, specifies only that the collection beclosedunder complementation and countable unions. However, these requirementsimmediately yield additional closure properties. The countably infinite version of DeMorgans s Laws of elementary set theory require that ifFi, i= 1,2,..are allmembers of a -field then so is: i=1Fi=[ i=1 Fci]cDr Conor McArdleEE414 - Probability & Stochastic Processes11/60 Event SpacesThus the -field properties imply that the collection of events in an event space isclosed under all set-theoretic operations (union, intersection, complementation,difference, etc.) so that performing set operations on events must result in otherevents inside the event closure requirement ensures that if we know the Probability of an eventAoccurring and Probability of an eventBoccurring, then we can also find the probabilityof logical combinations such as the Probability of bothAandBoccurring (intersectionof events), the Probability of eitherAandBoccurring (union of events), follows by similar set-theoretic arguments that any countable sequence of any of theset-theoretic operations (union, intersection, complementation, difference, symmetricdifference, etc.)

8 Performed on events in an event space must yield other events in theevent next turn to the question of how such event spaces may be Conor McArdleEE414 - Probability & Stochastic Processes12/60 Event Spaces: The Power SetPGiven a countable sample space , the collection of all subsets of is a -field (andthus a valid event space).This is true since any countable sequence of set-theoretic operations on subsets of must yield another subset of .Such a collection of all possible subsets of a sample space is called thePower SetPof the power set is the largest possible event space since it contains all subsets of .Note that, a finite sample space withnelements has a power set with at example, the power set of the binary sample space ={0,1}isP={{0},{1},{0,1}, } Conor McArdleEE414 - Probability & Stochastic Processes13/60 Event Spaces: -Fields Generated by a Family of EventsAlthough the power set of the sample space automatically yields a valid event space, itis possible to find a smaller event space, given some set of events of example, consider the experiment of tossing two coins together in a game wherewe are only interested in the event of tossing one head and one tail.

9 Denoting a headas 1 and a tail as 0, the appropriate sample space is: ={0,1}2={(0,0),(0,1),(1,0),(1,1)}The event space for the experiment can be defined as the power set of :P={{(0,0)},{(0,1)},{(1,0)},{(1,1)},{(0, 0),(0,1)},{(0,0),(1,0)},{(0,0),(1,1)},{( 0,1),(1,0)},{(0,1),(1,1)},{(0,1),(0,0)}, {(0,0),(0,1),(1,0)},{(0,0),(0,1),(1,1)}, {(0,0),(1,0),(1,1)},{(0,1),(1,0),(1,1)}, , }Can we find a smaller event space for this random experiment containing the event ofinterestA={(0,1),(1,0)}?Dr Conor McArdleEE414 - Probability & Stochastic Processes14/60 Event Spaces: -Fields Generated by a Family of EventsWe can in factgeneratethe smallest event space ( -field)Gthat our example, if we start with the event of interestA={(0,1),(1,0)}and applythe rules of the -field (all complements and countable unions are also in the field)iteratively we arrive at the event space:G={A,Ac,A Ac,A Ac}={{(0,1),(1,0)},{(0,0),(1,1)},{(0,1), (1,0),(0,0),(1,1)}, }We note that in this instance the chosen family of events of interest consisted of asingle eventA.

10 In general, the family may contain many give a more precise definition of a generated field we say that, given a family ofeventsAof interest, we may find the -fieldGgenerated byAby taking theintersection of all -fields on that containA, that is:G= { F:Fis a -field withA F}By this definition,Gmust be the smallest -field Conor McArdleEE414 - Probability & Stochastic Processes15/60 Event SpacesExercise 2 What is the power set of ={1,2,3,4}?Given ={1,2,3,4}, find the -field (event space) generated by the family ofeventsA={{1},{3,4}}.Although the notion of a generated -field has been introduced in the context of acountable sample space, it is more usual to take the power set as the de facto eventspace for countable sample spaces. Generated fields are most useful when definingevent spaces on uncountable sample spaces (for example the real line).In the uncountable case, a mathematical technicality arises with some subsets of thesample space ( some elements of the power set).