Cheat Sheet

1 Probability and Statistics Cheat Sheet Copyright c Matthias Vallentin, 2011. 6th March, 2011. This Cheat Sheet integrates a variety of topics in probability the- 12 Parametric Inference 11 20 Stochastic Processes 22. ory and statistics. It is based on literature [1, 6, 3] and in-class Method of Moments .. 11 Markov Chains .. 22. material from courses of the statistics department at the Univer- Maximum Likelihood .. 12 Poisson Processes .. 22. sity of California in Berkeley but also influenced by other sources Delta Method .. 12. [4, 5]. If you find errors or have suggestions for further topics , I 21 Time Series 23.

2 Multiparameter Models .. 12. would appreciate if you send me an email. The most recent ver- Stationary Time Series .. 23. Multiparameter Delta Method . 13. sion of this document is available at Estimation of Correlation .. 24. Parametric Bootstrap .. 13 Non-Stationary Time Series .. 24. To reproduce, please contact me. Detrending .. 24. 13 Hypothesis Testing 13 ARIMA models .. 24. Contents 14 Bayesian Inference 14. Causality and Invertibility .. 25. Spectral Analysis .. 25. 1 Distribution Overview 3 Credible Intervals .. 14. Discrete Distributions .. 3 Function of Parameters.

3 14 22 Math 26. Continuous Distributions .. 4 Priors .. 15 Gamma Function .. 26. Conjugate Priors .. 15 Beta Function .. 26. 2 Probability Theory 6 Bayesian Testing .. 15 Series .. 27. Combinatorics .. 27. 3 Random Variables 6 15 Exponential Family 16. Transformations .. 7. 16 Sampling Methods 16. 4 Expectation 7 The Bootstrap .. 16. Bootstrap Confidence Intervals . 16. 5 Variance 7. Rejection Sampling .. 17. 6 Inequalities 8 Importance Sampling .. 17. 7 Distribution Relationships 8 17 Decision Theory 17. Risk .. 17. 8 Probability and Moment Generating Admissibility.

4 17. Functions 9 Bayes Rule .. 18. Minimax Rules .. 18. 9 Multivariate Distributions 9. Standard Bivariate Normal .. 9 18 Linear Regression 18. Bivariate Normal .. 9 Simple Linear Regression .. 18. Multivariate Normal .. 9. Prediction .. 19. 10 Convergence 9 Multiple Regression .. 19. Law of Large Numbers (LLN) .. 10 Model Selection .. 19. Central Limit Theorem (CLT) .. 10. 19 Non-parametric Function Estimation 20. 11 Statistical Inference 10 Density Estimation .. 20. Point Estimation .. 10 Histograms .. 20. Normal-based Confidence Interval .. 11 Kernel Density Estimator (KDE) 21.

5 Empirical Distribution Function .. 11 Non-parametric Regression .. 21. Statistical Functionals .. 11 Smoothing Using Orthogonal Functions 21. 1 Distribution Overview Discrete Distributions Notation1 FX (x) fX (x) E [X] V [X] MX (s).. 0 x<a (b a + 1)2 1 eas e (b+1)s . bxc a+1 I(a < x < b) a+b Uniform Unif {a, .. , b} a x b b a b a+1 2 12 s(b a). 1 x>b . Bernoulli Bern (p) (1 p)1 x px (1 p)1 x p p(1 p) 1 p + pes ! n x Binomial Bin (n, p) I1 p (n x, x + 1) p (1 p)n x np np(1 p) (1 p + pes )n x k k !n n! x X X. Multinomial Mult (n, p) px1 1 pkk xi = n npi npi (1 pi ) pi e si x1 !

6 Xk ! i=1 i=0. ! m m x . x np x n x nm nm(N n)(N m). Hypergeometric Hyp (N, m, n) N. N/A. N 2 (N 1). p . np(1 p) x N. ! r x+r 1 r 1 p 1 p p Negative Binomial NBin (n, p) Ip (r, x + 1) p (1 p)x r r r 1 p p2 1 (1 p)es 1 1 p p Geometric Geo (p) 1 (1 p)x x N+ p(1 p)x 1 x N+. p p2 1 (1 p)es x X i x e s Poisson Po ( ) e e (e 1). i=0. i! x! Uniform (discrete) Binomial Geometric Poisson n = 40, p = p = =1.. n = 30, p = p = =4. n = 25, p = p = = 10. 1. PMF. PMF. PMF. PMF.. n .. a b 0 10 20 30 40 0 2 4 6 8 10 0 5 10 15 20. x x x x 1 We use the notation (s, x) and (x) to refer to the Gamma functions (see ), and use B(x, y) and Ix to refer to the Beta functions (see ).

7 3. Continuous Distributions Notation FX (x) fX (x) E [X] V [X] MX (s).. 0 x<a (b a)2 esb esa . x a I(a < x < b) a+b Uniform Unif (a, b) a<x<b b a b a 2 12 s(b a). 1 x>b . (x )2. Z x 2 s2.. 1. N , 2 2.. Normal (x) = (t) dt (x) = exp exp s +. 2 2 2 2. (ln x )2.. 1 1 ln x 1 2 2 2. ln N , 2 e + /2. (e 1)e2 + .. Log-Normal + erf exp . 2 2 2 2 x 2 2 2 2.. 1 T. 1 (x ) 1. Multivariate Normal MVN ( , ) (2 ) k/2 | | 1/2 e 2 (x ) exp T s + sT s 2. ( +1)/2. +1.. 2 x2. Student's t Student( ) Ix , . 1+ 0 0. 2 2 2 .. 1 k x 1. Chi-square 2k , xk/2 e x/2 k 2k (1 2s) k/2 s < 1/2.

8 (k/2) 2 2 2k/2 (k/2). r d (d1 x)d1 d2 2. 2d22 (d1 + d2 2).. d1 d1 (d1 x+d2 )d1 +d2 d2. F F(d1 , d2 ) I d1 x , d1 d1 d2 2 d1 (d2 2)2 (d2 4).. d1 x+d2 2 2 xB 2. , 2. 1 x/ 1. Exponential Exp ( ) 1 e x/ e 2 (s < 1/ ). 1 s . ( , x/ ) 1 1. Gamma Gamma ( , ) x 1 e x/ 2 (s < 1/ ). ( ) ( ) 1 s , x . 1 /x 2 2( s) /2 p . Inverse Gamma InvGamma ( , ) x e >1 >2 K 4 s ( ) ( ) 1 ( 1)2 ( 2)2 ( ). P . k i=1 i Y 1. k i E [Xi ] (1 E [Xi ]). Dirichlet Dir ( ) Qk xi i Pk Pk i=1 ( i ) i=1 i=1 i i=1 i + 1. k 1. ! ( + ) 1 X Y +r sk Beta Beta ( , ) Ix ( , ) x (1 x) 1 1+. ( ) ( ) + ( + )2 ( + + 1) r=0.

9 + +r k! k=1.. sn n .. k k x k 1 (x/ )k 1 2 X n . Weibull Weibull( , k) 1 e (x/ ) e 1 + 2 1 + 2 1+. k k n=0. n! k x . m x xm x . Pareto Pareto(xm , ) 1 x xm m +1 x xm >1 m >2 ( xm s) ( , xm s) s < 0. x x 1 ( 1)2 ( 2). 4. Uniform (continuous) Normal Log normal Student's t =1. = 0, 2 = = 0, 2 = 3. = 0, 2 = 1 = 2, 2 = 2 =2. = 0, 2 = 5 = 0, 2 = 1 =5. = . = 2, 2 = = , 2 = 1. = , 2 = 1. = , 2 = 1. PDF. PDF. PDF. (x). 1.. b a . a b 4 2 0 2 4 4 2 0 2 4. x x x x 2. F Exponential Gamma k=1 d1 = 1, d2 = 1 =2 = 1, = 2. k=2 d1 = 2, d2 = 1 =1 = 2, = 2. k=3 d1 = 5, d2 = 2 = = 3, = 2.

10 K=4 d1 = 100, d2 = 1 = 5, = 1. k=5 d1 = 100, d2 = 100 = 9, = PDF. PDF. PDF. PDF. 0 2 4 6 8 0 1 2 3 4 5 0 1 2 3 4 5 0 5 10 15 20. x x x x Inverse Gamma Beta Weibull Pareto = 1, = 1 = , = = 1, k = xm = 1, = 1. = 2, = 1 = 5, = 1 = 1, k = 1 xm = 1, = 2. = 3, = 1 = 1, = 3 = 1, k = xm = 1, = 4. = 3, = = 2, = 2 = 1, k = 5. = 2, = 5. 4. 3. 3. 2. PDF. PDF. PDF. PDF. 2. 1. 1. 0. 0. 0 1 2 3 4 5 0 1 2 3 4 5. x x x x 5. 2 Probability Theory Law of Total Probability n n Definitions X G. P [B] = P [B|Ai ] P [Ai ] = Ai Sample space i=1 i=1. Outcome (point or element) Bayes' Theorem Event A.