Chap. 5: Joint Probability Distributions

1 Chap. 5: Joint Probability Distributions Probability modeling of several RV s We often study relationships among variables. Demand on a system = sum of demands from subscribers (D = S1 + S2 + . + Sn). Surface air temperature & atmospheric CO2. Stress & strain are related to material properties; random loads; etc. Notation: Sometimes we use X1 , X2 , ., Xn Sometimes we use X, Y, Z, etc. 1. Sec : Basics First, develop for 2 RV (X and Y). Two Main Cases I. Both RV are discrete II. Both RV are continuous I. (p. 185). Joint Probability Mass Function (pmf) of X and Y is defined for all pairs (x,y) by p( x, y ) P( X x and Y y ). P( X x, Y y ). 2. pmf must satisfy: p( x, y) 0 for all ( x, y). x y p( x, y) 1. for any event A, P ( X , Y ) A p( x, y). ( x , y ) A. 3. Joint Probability Table: Table presenting Joint Probability distribution : y Entries: p( x, y ) 1 2 3. P(X = 2, Y = 3) = .13 x 1 .10 .15 .22. 2 .30 .10 .13. P(Y = 3) = .22 + .13 = .35. P(Y = 2 or 3) = .15 + .10 + .35 =.60. 4. The marginal pmf X and Y are p X ( x) y p( x, y) and pY ( y) x p( x, y).

2 Y 1 2 3. x 1 .10 .15 .22 .47. 2 .30 .10 .13 .53..40 .25 .35. x 1 2 y 1 2 3. pX(x) .47 .53 pY(y) .40 .25 .35. 5. II. Both continuous (p. 186). A Joint Probability density function (pdf) of X and Y is a function f(x,y) such that f(x,y) > 0 everywhere .. f ( x, y)dxdy 1. and P[( X , Y ) A] f ( x, y)dxdy A. 6. pdf f is a surface above the (x,y)-plane A is a set in the (x,y)-plane. P[( X , Y ) A] is the volume of the region over A under f. (Note: It is not the area of A.). f y x A. 7. Ex) X and Y have Joint PDF. f(x,y) = c x y2 if 0 < x < y < 1. = 0 elsewhere. Find c. First, draw the region where f > 0. 1 y 1. 1 cxy2. dxdy . y 0 0 0 1. 1 1. x cxy2. dydx 1 y cxy 2. 0 x (not dxdy x 0. 1 y 1 1. dy c / 10. 2 2 2 y 4. cxy dxdy c y [.5 x | 0 ]dy c .5 y 0 0 0 0. so, c = 10 1. Find P(X+Y<1). y First, add graph of x + y =1. 0 1..5 y 1 1 y x P( X Y 1) .. 10 xy 2. dxdy 10 xy 2. dxdy 0 0 .5 0..5 1 x 1 x .5. y 3. x 10 xy dydx 10 0 x 3 dx . 2. 0 x .5. (10 / 3) x((1 x) 3 x 3 )dx .135. 0. Marginal pdf (p.)

3 188) . Marginal pdf of X: f X ( x) f ( x, y)dy .. Marginal pdf of Y: fY ( y ) .. f ( x, y)dx Ex) X and Y have Joint PDF. f(x,y) = 10 x y2 if 0 < x < y < 1 , and 0 else. For 0 < y < 1: y y fY ( y ) f ( x, y)dx 10 xy dx 10 y xdx 5 y 2 2 4. 0 0. and fY ( y) 0 otherwise. 10. marginal pdf of Y: fY ( y) 5 y for 0 y 1 and is 0 otherwise. 4. marginal pdf of Y: you check f X ( x) (10 / 3) x(1 x ) for 0 x 1. 3. Notes: and is 0 otherwise. 1. x cannot appear in fY ( y ) (y can t be in f X (x) ). 2. You must give the ranges; writing fY ( y) 5 y 4. is not enough. Math convention: writing Y f ( y ) 5 y 4. with no range means it s right for all y, which is very wrong in this example. 11. Remark: distribution Functions For any pair of jointly distributed RV, the Joint distribution function (cdf) of X and Y. is F ( x, y) P( X x, Y y). defined for all (x,y). For X,Y are both continuous: 2. f ( x, y ) F ( x, y ). x y wherever the derivative exists. 12. Extensions for 3 or more RV: by example X, Y, Z are discrete RV with Joint pmf p( x, y, z) P( X x, Y y, Z z).

4 Marginal pmf of X is p X ( x) y z p( x, y, z ) (= P(X = x)). ( Joint ) marginal pmf of X and Y is p XY ( x, y) z p( x, y, z ) (= P(X=x, Y = y)). 13. X, Y, Z are continuous RV with Joint pdf f(x,y,z): marginal pdf of X is . f X ( x) f ( x, y, z )dydz . ( Joint ) marginal pmf of X and Y is . f XY ( x, y ) f ( x, y, z )dz . 14. Conditional Distributions & Independence .. Marginal pdf of X: f X ( x) f ( x, y )dy . Marginal pdf of Y: . fY ( y ) f ( x, y)dx . Conditional pdf of X. given Y=y (h(y) > 0) f ( x | y) f ( x, y) / h( y). Conditional prob P( X A | Y y) f ( x | y)dx for X for y fixed A. 15. Conditional Distributions & Independence Review from Chap. 2: For events A & B where P(B) > 0, define P(A|B). to be the conditional prob. that A occurs given B occurred: P(A | B)=P(A I B) / P(B). Multiplication Rule: P(A I B) = P(A) P(B|A). = P(B) P(A|B). Events A and B are independent if P(B|A) = P(B). or equivalently P(A I B) = P(A) P(B). Extensions to RV. Again, first, develop for 2 RV (X and Y).

5 Two Main Cases I. Both RV are discrete II. Both RV are continuous I. (p. 193). Conditional Probability Mass Function (pmf) of Y given X = x is p( x, y ) Joint pY | X ( y | x) . p X ( x) marginal of condition as long as p X ( x) 0. 17. Note that idea is the same as in Chap. 2. P( X x, Y y ). P(Y y | X x) . P( X x). as long as P( X x) 0. However, keep in mind that we are defining a (conditional) prob. dist for Y for a fixed x 18. Example: y x 1 2. 1 2 3 pX(x) .47 .53. x 1 .10 .15 .22 .47. y 1 2 3. 2 .30 .10 .13 .53..40 .25 .35 pY(y) .40 .25 .35. Find cond l pmf of X given Y = 2: p( x, y ) gives p ( x | 2) p( x,2). p X |Y ( x | y ) X |Y. pY ( y ) pY (2). So x 1 2. pX|Y(x|2) .15/.25=.60 .10/.25=.40. 19. II. Both RV are continuous (p. 193). Conditional Probability Density Function (pdf) of Y given X = x is f ( x, y) Joint pdf f Y | X ( y | x) . f X ( x) marginal pdf of condition as long as f X ( x) 0. The point: P(Y A | X x) fY | X ( y | x)dy A. 20. Remarks ALWAYS: for a cont. RV, prob it s in a set A is the integral of its pdf over A: no conditional; use the marginal pdf with a condition; use the right cond l pdf Interpretation: For cont.

6 X, P(X = x) = 0, so by Chap 2 rules, P(Y A | X x) is meaningless. There is a lot of theory that makes sense of this For our purposes, think of it as an approximation to P(Y A | X x). that is given X lies in a tiny interval around x . 21. Ex) X, Y have pdf f(x,y) = 10 x y2 if 0 < x < y < 1 , and = 0 else. Conditional pdf of X given Y= y: f X |Y ( x | y) f ( x, y) / fY ( y). We found fY(y) = 5y4 for 0<y<1, = 0 else. So 2. 10 xy f ( x | y) 4. if 0 < x < y < 1. 5y Final Answer: For a fixed y, 0 < y < 1, fX|Y(x | y) = 2x / y2 if 0 < x < y, and = 0 else. 22. ( f(x | y) = 2x / y2 0 < x < y, and = 0 else. )..2. 2 x / .3. 2. P(X < .2 | Y = .3) dx 0. P(X < .35 | Y = .3) = 1. x FX |Y ( x | y) 2t / y dt x / y 2 2 2. if 0 < x < y 0. 1. FX|Y(x|y). y 1. 23. Last Time: X, Y have pdf f(x,y).. Marginal pdf of X: f X ( x) f ( x, y )dy . Marginal pdf of Y: . fY ( y ) f ( x, y)dx . Conditional pdf of X. given Y=y (fY(y) > 0) f X |Y ( x | y) f ( x, y) / fY ( y). Conditional prob P( X A | Y y ) f X |Y ( x | y)dx for X for y fixed A.

7 24. Three More Topics 1. Multiplication Rule for pdf: f ( x, y) f X |Y ( x | y) fY ( y) fY | X ( y | x) f X ( x). [For events P(A I B) = P(A) P(B|A) = P(B) P(A|B) ]. Extension, by example: f ( x, y, z) f X ( x) fY | X ( y | x) f Z | XY ( z | x, y). [Chap 2: P(A I B I C) = P(A) P(B|A) P(C|A I B) ]. 25. 2. Independence Chap. 2: A, B are independent if P(A|B)=P(A). or equivalently, P(A I B) = P(A)P(B). X and Y are independent RV if and only if f X |Y ( x | y) f X ( x). for all (x,y) for which f(x,y)>0, or f ( x, y) f X ( x) fY ( y). for all (x,y) for which f(x,y)>0. , the Joint is the product of the marginals. 26. 2. Independence More general: X1, X2, ., Xn are independent if for every subset of the n variable, their Joint pdf is the product of their marginal pdf s. f ( x1 , x2 ,.., xn ) f1 ( x1 ) f 2 ( x2 ).. f n ( xn ). and f ( x1 , x7 , x28 ) f1 ( x1 ) f 7 ( x7 ) f 28 ( x28 ) etc. 27. 3. Bayes Thm P( A | Br ) P( Br ). Chap. 2: P( Br | A) .. k i 1. P ( A | Bi ) P ( Bi ). Bayes Theorem for Disc RV s: For p X ( x) 0.

8 P ( x, y ) p ( x, y ). pY | X ( y | x) . p X ( x ) p ( x, y ). y Note: pmf s are prob s, so this is Bayes s Thm in disc RV notation 28. 3. Bayes Thm p ( x, y ) p ( x, y ). pY | X ( y | x) . p X ( x) p ( x, y ). y Bayes Theorem for Cont RV s: For f X ( x) 0. f ( x, y ) f ( x, y ). fY | X ( y | x) . f X ( x).. f ( x, y)dy Note: pdf s are not prob s but the formula works 29. x Ex) X, Y have PDF f ( x, y) c( x y )e 2 2. if 0 x , x y x and 0 elsewhere. Find the conditional PDF of Y given X=x: fY | X ( y | x) f ( x, y) / f X ( x). x f X ( x) c( x 2 y 2 )e x dy ce x [ x 2 y y 3 / 3 | x x x 3 x (4c / 3) x e ,0 x . means c=1/8 (we ll see why later)). 30. hence x ( x y )e 2 2. fY | X ( y | x) (3 / 4) 3 x , x y x xe and 0 elsewhere Partial check, integrate this and verify we get 1. (x y ). x x 2 2.. x fY | X ( y | x)dy (3 / 4) . x x 3. dy (3 / x 4)[ x y y / 3 | 1. 3 2 3 x x . Don t need c f ( y | x ) f ( x, y ) / f ( x, y)dy . Conditional Independence & Learning (beyond the text). Ex) Very large population of people.]]

9 Let Y be the unknown proportion having a disease (D). Sample 2 people at random , without replacement &. check for D. Define X1 = 1 if person 1 has D, X1 =0 if not. Define X2 for person 2 the same way. Note: the X s are discrete, Y is continuous 32. Model assumptions 1. Given Y= y, X1 and X2 are conditionally independent, with PD s 1 x1. p1 ( x1 | y) y (1 y) , x1 0,1. x1. 1 x p2 ( x2 | y) y (1 y) , x2 0,1. x 2 2. Hence, p12 ( x1 , x2 | y) p1 ( x1 | y) p2 ( x2 | y). x1 x2 2 ( x1 x2 ). y (1 y) , x1 & x2 0,1. 2. Suppose we know little about Y: Assume fY(y)=1, 0<y<1, and 0 elsewhere. 33. Learn about Y after observing X1 & X2? Answer fY | X1X 2 ( y | x1 , x2 ) p12 ( x1 , x2 | y) fY ( y) / p12 ( x1 , x2 ). 1. where p12 ( x1 , x2 ) y x1 x2. (1 y ) 2 ( x1 x2 ). dy, 0. x1 & x2 0,1. Note: X1 & X2 are unconditionally dependent. 34. Learn about Y after observing X1 & X2? Answer fY | X1X 2 ( y | x1 , x2 ) p12 ( x1 , x2 | y) fY ( y) / p12 ( x1 , x2 ). Ex) Observe X1 = X2 =1. Then 1. p12 (1,1) y 2 dy 1 / 3.

10 0. 2. y and f ( y | 1,1) 1. 3 y ,0 y 1. 2. y dy 2. 0 35. Summary Before data, we had no idea about Y, our prior pdf was fY(y)=1, 0<y<1. After seeing 2 out of 2 people sampled have D, we update to the posterior pdf fY(y|1,1) = 3y2, 0<y<1. pdf 3 fY(y|1,1). 1 fY(y). 0 1. y 36. Sect. Expected Values (p. 197) For discrete RV s X, Y with Joint pmf p, the expected value of h(X,Y) is E[h( X , Y )] x y h( x, y) p( x, y). if finite. For continuous RV s X, Y with Joint pdf f, the expected value of h(X,Y) is . E[h( X , Y )] h( x, y ) f ( x, y )dxdy . if finite. 37. Ex) X and Y have pdf f(x,y) = x + y, 0 < x < 1, 0 < y < 1, and 0 else. Find E(XY2). 1 1. E ( XY ) xy ( x y )dxdy 17 / 72. 2 2. 0 0. (Check my integration). Extensions, by example: . E[h( X , Y , Z )] .. h( x, y, z ) f ( x, y, z )dxdydz 38. Important Result: If X 1 , X 2 , , X n are independent RV' s, then E[h1 ( X 1 )h2 ( X 2 ) hn ( X n )] . E[h1 ( X 1 )]E[h2 ( X 2 )] E[hn ( X n )]. Under independence, the expectation of product = product of expectations.