Chapter 3: Basics from Probability Theory and Statistics

1 IRDM WS 2015 Chapter 3: Basics from Probability Theoryand Probability TheoryEvents, Probabilities, Bayes Theorem,Random Variables, Distributions, Moments, Tail Bounds, Central Limit Theorem, Entropy Statistical InferenceSampling, Parameter Estimation, Maximum Likelihood,Confidence Intervals, Hypothesis Testing, p-Values, Chi-Square Test, Linear and Logistic RegressionmostlyfollowingL. Wasserman Chapters6, 9, 10, 13 IRDM WS Statistical InferenceA statisticalmodelisa setofdistributions(orregressionfunctions ), , all unimodal, smooth parametricmodelisa setthatiscompletelydescribedbya finite numberofparameters,( , thefamilyofNormal distributions).Statistical inference: givena sample X1, .., Xnhowdo weinferthedistributionoritsparameterswit hina modelswithonespecific outcome(response) variable Y, thisiscalledpredictionorregression, fordiscreteoutcomevariable also (x) = E[Y | X=x] : biomedicalmarkers cancerornotExampleforregression: businessindicators stock priceIRDM WS 2015 Sampling Illustrated3-41 Distribution X(populationofinterest)Samples X1, X2.

2 , XnStatistical Inference:WhatcanwesayaboutXbasedon X1, X2, .., Xn?Example: estimatetheaveragesalaryin Germany?Approach 1: askyour10 neighborsApproach 2: ask100 randompeopleyouspoton theInternetApproach 2: askall 1000 livingGermans in WikipediaApproach 4: ask1000 randompeoplefromall agegroups, jobs, .. IRDM WS 2015 Basic TypesofStatistical Inference3-42 Given: independentandidenticallydistributed(iid ) samplesX1, X2, .., Xnfrom(unknown) distributionX Parameter estimation: Confidenceintervals: Hypothesis testing: Regression (forparameterfitting)Whatistheparameterp ofa Bernoulli coin?Whatarethevaluesof and ofa Normal distribution?Whatare 1, 2, 1, 2ofa Poissonmixture?Whatistheinterval[mean tolerance] theexpectationofmyobservationsormeasurem entsfalls intotheintervalwithhigh confidence?H0: p=1/2 (fair coin) vs. H1: p 1/2 H0: p1 = p2 (methodshavesame precision) vs. H1: p1 p2 IRDM WS Statistical Parameter EstimationA point estimatorfor a parameter of a prob.

3 Distribution is arandom variable X derived from a random sample X1, .., :Sample mean:Sample variance: niiXn:X1121211)XX(n:Snii An estimatorT forparameter isunbiasedif; estimatoron a sample ofsizen isconsistentif ]T[E01 eachfor]T[Plimn ]T[ESample mean and sample variance are unbiased, consistent estimators with minimal WS 2015 EstimationErrorLet = T( ) be an estimator for parameter over sample X1, .., Xn. The distribution of is called the sampling standard errorfor is:n n n The mean squared error (MSE)for is: n 2n MSE( ) E[() ] 2nn bias () Var[] If bias 0 and se 0 then the estimator is estimator is asymptotically Normalifconverges in distribution to standard Normal N(0,1) n n () / se 3-44 = ( )IRDM WS 2015 Nonparametric EstimationThe empiricaldistributionfunctionisthecdftha tputsprob. mass1/n at eachdatapointXi:whereindicatorfunctionI( )is1 if and0 otherwisen Fnnii11 F ( x )I( Xx )n A statistical functionalT(F) is any function of F, , mean, variance, skewness, median, quantiles, correlationThe plug-in estimatorof = T(F) is: nn T ( F ) 3-45 IRDM WS 2015 NonparametricEstimation: HistogramsInsteadofthefullempiricaldistr ibution, oftencompactdatasynopsesmaybeused, such ashistogramswhereX1.

4 , Xnaregroupedintom cells(bucketsorbins) c1, .., cm withbucketboundarieslb(ci) andub(ci) (c1) = , ub(cm) = , ub(ci) = lb(ci+1) for1 i<m, andfreq(ci) = nnii11 F ( x )I( lb( c )Xub( c ))n Histogramsprovidea (discontinuous) :X1= X2= 1X3= X4= X5= 2X6= .. X10= 3X11= .. X14= 4X15= .. X17= 5X18= X19= 6X20= 7 IRDM WS 20153-47 Kinds ofHistogramsequidistantbucketsnon-equidi stantbucketsIRDM WS 2015 MethodofMomentsMethod-of-moments estimatorsareusuallyconsistentandasympot icallyNormal, but maybebiased3-48 Suppose parameter = ( 1, .., k) has kcomponents Compute j-thmomentfor 1 j k: Compute j-thsample momentfor 1 j k: Method-of-momentsestimate of is obtained by solving a system of kequations in kunknowns:IRDM WS 2015 Example: MethodofMomentsLetX1, .., Xn~ Normal( , 2)3-49 1= = 2= 2= + 2= 2+ 2 Solvetheequationsystem: = 1= 1=1 =1 2+ 2= 2= 2=1 =1 2 Solution: =1 =1 = 2=1 =1 2 IRDM WS 2015 Parametric Inference:Maximum Likelihood Estimators (MLE)Estimateparameter ofa postulateddistributionf( ,x)such thattheprobabilitythatthedataofthesample aregeneratedbythisdistributionismaximize d.

5 Maximum likelihoodestimation:MaximizeL(x1,..,xn, ) = P[x1, .., xnoriginatefromf( ,x)]oftenwrittenas = L( , x1,..,xn)= = ( ,, )ormaximizelog Lifanalyticallyuntractable usenumericaliterationmethods3-50 IRDM WS 2015 MLE PropertiesMaximum Likelihood Estimators areconsistent, asymptotically Normal, andasymptotically optimalin the following sense:Consider two estimators U and T which are asymptotically u2and t2denote the variances of the two Normal distributionsto which U and T converge in asymptotic relative efficiencyof U to T is ARE(U,T) = t2/u2 .Theorem:For an MLE and any other estimatorthe following inequality holds: n n nn ARE(,) 1 3-51 IRDM WS 2015 Simple Example forMaximum Likelihood Estimatorgiven: coin with Bernoulli distribution with unknown parameter p f r head, 1-p for tail sample (data): k times head with n coin tossesneeded: maximum likelihood estimation of pLet L(k, n, p) = P[sample is generated from distr.]

6 With param. p]knkppkn )1(Maximizelog-likelihoodfunctionlog L (k, n, p):nlog L logk log p(n k) log (1 p)k nkp 01log pknpkpL3-52 IRDM WS 2015 Advanced Example for Maximum Likelihood Estimatorgiven: Poisson distribution with parameter (expectation) sample (data): numbers x1, .., xn N0needed: maximum likelihood estimation of 01ln0 riiifL xxnffiniiriirii 1001 riifin!ie),x,..,x(L01 Let r be the largest among these numbers, and let f0, .., frbe the absolute frequencies of numbers 0, .., WS 2015 Sophisticated Example for Maximum Likelihood Estimatorgiven: discrete uniform distribution over [1, ] N0and density f(x) = 1/ sample (data): numbers x1, .., xn N0 MLE for is max{x1, .., xn} (see Wasserman p. 124)3-54 IRDM WS 2015 MLE for Parameters of Normal Distributions ni)ix(nne),,x,..,x(L12222121 ni2i1ln( L )12( x)02 021212422 nii)x(n)Lln( niixn 11 2121) x(n nii 3-55 IRDM WS 2015 Analytically Non-tractable MLE for parametersof multivariate Normal Mixture).

7 ,,,..,,,..,,(111kkkxf kjjxjTjxjmje1)(1)(21)2(1 with expectation values and invertible, positive definite, symmetricm m covariance matrices j j kjjjjxn1),,( maximize log-likelihood function: nikjjjijniinxnxPxxL1111),,(log]|[log:),, ..,(log consider samples from a mixture of multivariate Normal distributionswith the density ( height and weight of males and females):3-56 IRDM WS 2015 Expectation-MaximizationMethod(EM)WhenL( , X1, .., Xn) isanalyticallyintractablethen introducelatent (non-observable) randomvariable(s) Z such that:jointdistributionJ(X1, .., Xn, Z, ) of complete dataistractable iterativelycompute: Expectation(E Step): computeexpectedcompletedatalikelihoodEZ [log J(X1, .., Xn, Z | (t))] givena previousestimateof Maximization(M Step): estimate (t+1)thatmaximizesEZ [log J(X1, .., Xn, Z | (t))] 2-57detailsdependon distributionat hand(oftenmixturemodels)convergenceguara nteed, but problemisnon-convex numericalmethodsIRDM WS 2015 Bayesian Viewpoint of Parameter Estimation assume prior distribution g( )of parameter choose statistical model (generative model) f (x | )that reflects our beliefs about RV X given RVs X1.)

8 , Xnfor observed data, the posterior distribution is h ( | x1, .., xn)for X1=x1, .., Xn=xnthe likelihood iswhich implies(posterior is proportional tolikelihood times prior)MAP estimator (maximum a posteriori):compute that maximizes h ( | x1, .., xn)given a prior for )(g), (L~) |(hn1n1 n1i'iin1iin1)(g)'(g)'|x(f)x|(h)|x(f), (L 2-58 IRDM WS ConfidenceIntervalsEstimator T for an interval for parameter such thatFor the distribution of random variable X a value x (0< <1) withis called a quantile; the quantile is called the the normal distribution N(0,1) the quantile is denoted . 1]xX[P]xX[P 1]aTaT[P[T-a, T+a] istheconfidenceintervaland1- : (a)= a= IRDM WS 2015 Confidence Intervals for Expectations (1)Let x1, .., xnbe a sample from a distribution with unknownexpectation and known variance 2. For sufficiently large n the sample mean is N( , 2/n) distributedand is N(0,1) distributed:X n)X( 1)z(2))z(1()z()z()z(]zn)X(z[P ]nzXnzX[P 12121]nXnX[P//),(Nofquantile)(:z1021 thensettodetermineintervalForrequiredcon fidenceintervalorconfidencelevel1- set]aX,aX[ na:z orthenlookup (z)todetermine1 /2za:n 3-60 IRDM WS 2015 Normal Distribution Table3-61 IRDM WS 2015 Confidence Intervals for Expectations (2)Let x1.

9 , xnbe a sample from a distribution with unknownexpectation and unknown variance 2and sample variance sufficiently large n the random variableSn)X(:T has a t distribution (Student distribution)with n-1 degrees of freedom:21211221 nn,Tntnnn)t(f with the Gamma function: 010xf rdtte)x(xt))x(x)x(and)(propertiesthewith ( 111 1211211]nStXnStX[P/,n/,n3-62 IRDM WS 2015 Student s t Distribution Table3-63 William Gosset(1876-1937)A. Student:The Probable Error of a Mean, Biometrika6(1), 1908forinterval[ , + ]:thenlookup (z)todetermine1 /2 IRDM WS 2015 Example: ConfidenceIntervalforExpectation3-64X: time forstudenttosolveexercisen=16 samples, = , 2= ) Assume 2isknown: 2= A1) Estimate ) Estimate with1 = confidenceB) Assume 2isunknownB1) Estimate ) Estimate with1 = confidence1)z(2))z(1()z()z()z(]zn)X(z[P ]nzXnzX[P 12121]nXnX[P// na:z ),(Nofquantile)(:z1021 forconfidence1 :thensettodetermineintervalza:n IRDM WS Hypothesis TestingHypothesis testing: aimstofalsifysomehypothesisbylack ofstatisticalevidence design oftestRV (teststatistic) andits(approx.

10 / limit) distribution3-65 Toss a coin n times and judge if the coin is fairX1, .., Xn~ Bernoulli(p), coin is fair if p = Let the null hypothesis H0be the coin is fair The alternative hypothesis H1is then the coin is not fair Intuitively, if is large, we should reject H0 Example:H0isdefault, interestisin H1: aimtorejectH0( suspectingthatthecoinisunfair)IRDM WS 2015 Hypothesis TestingTerminology(1)A hypothesistestdeterminesa probability1- (testlevel , significancelevel) thata sample X1, .., Xnfromsomeunknownprobabilitydistribution hasa :1)The sample originatesfroma normal )Undertheassumptionofa normal distributionthesample originatesfroma N( , 2) )Tworandomvariables )Tworandomvariables )Parameter ofa Poissondistributionfromwhichthesample form :null hypothesis H0vs. alternative hypothesis H1needstestvariable (teststatistic) X (derivedfromX1.