Advanced statistical methods for data analysis – Lecture 1

1 1 Glen CowanMultivariate statistical methods in Particle PhysicsAdvanced statistical methods for data analysis Lecture 1 Glen CowanRHUL ~cowanUniversit t Mainz Klausurtagung des GK Eichtheorien exp. Bullay/Mosel15 17 September, 20082 Glen CowanMultivariate statistical methods in Particle PhysicsOutlineMultivariate methods in particle physicsSome general considerationsBrief review of statistical formalismMultivariate classifiers:Linear discriminant functionNeural networksNaive Bayes classifierKernel based methodsk Nearest NeighbourDecision treesSupport Vector Machines3 Glen CowanMultivariate statistical methods in Particle PhysicsResources on multivariate methods Bishop, Pattern Recognition and Machine Learning, Springer, 2006T. Hastie, R. Tibshirani, J. Friedman, The Elements of statistical Learning, Springer, 2001R.

2 Duda, P. Hart, D. Stork, Pattern Classification, 2nd ed., Wiley, 2001A. Webb, statistical Pattern Recognition, 2nd ed., Wiley, 2002 Materials from some recent meetings:PHYSTAT conference series (2002, 2003, 2005, 2007,..) see workshop on multivariate analysis , 11 February, Lectures on Machine Learning by Ilya Narsky (2006) :4 Glen CowanMultivariate statistical methods in Particle PhysicsSoftwareTMVA, H cker, Stelzer, Tegenfeldt, Voss, Voss, physics/0703039 StatPatternRecognition, I. Narsky, physics/0507143 Further info from ~ wide variety of methods , many complementary to TMVAC urrently appears project no longer to be supportedFrom , also distributed with ROOTV ariety of classifiersGood manual5 Glen CowanMultivariate statistical methods in Particle PhysicsData analysis at the LHCThe LHC experiments are expensive~ $1010 (accelerator and experiments)the competition is intense(ATLAS vs.)

3 CMS) vs. Tevatronand the stakes are high:4 sigma effect5 sigma effectSo there is a strong motivation to extract all possible informationfrom the Glen CowanMultivariate statistical methods in Particle PhysicsThe Large Hadron ColliderCounter rotating proton beamsin 27 km circumference ringpp centre of mass energy 14 TeVDetectors at 4 pp collision points:ATLASCMSLHCb (b physics)ALICE (heavy ion physics)general purpose7 Glen CowanMultivariate statistical methods in Particle PhysicsThe ATLAS detector2100 physicists37 countries 167 universities/labs25 m diameter46 m length7000 tonnes~108 electronic channels8 Glen CowanMultivariate statistical methods in Particle PhysicsA simulated SUSY event in ATLAS high pTmuonshigh pT jets of hadronsmissing transverse energypp9 Glen CowanMultivariate statistical methods in Particle PhysicsBackground eventsThis event from Standard Model ttbar production alsohas high pT jets and muons,and some missing transverseenergy.

4 Can easily mimic a SUSY Glen CowanMultivariate statistical methods in Particle PhysicsStatement of the problemSuppose for each event we measure a set of numbers x= x1, ,xn x1 = jet pT x2 = missing energyx3 = particle measure, .. xfollows some n dimensional joint probability density, which depends on the type of event produced, , was itpp t t,pp g g, xixjWe want to separate (classify) the event types in a way that exploits the information carried in many hypotheses (class labels) H0, H1, .. Often simply signal , background p x H0 p x H1 11 Glen CowanMultivariate statistical methods in Particle PhysicsFinding an optimal decision boundaryMaybe select events with cuts :xi < cixj < cjOr maybe use some other type of decision boundary:Goal of multivariate analysis is to do this in an optimal Glen CowanMultivariate statistical methods in Particle PhysicsGeneral considerationsIn all multivariate analyses we must consider of variables to useFunctional form of decision boundary (type of classifier)Computational issuesTrade off between sensitivity and complexityTrade off between statistical and systematic uncertaintyOur choices can depend on goals of the analysis , ,Event selection for further studySearches for new event types13 Glen CowanMultivariate statistical methods in Particle PhysicsProbability quick reviewFrequentist (A = outcome of repeatable observation):Subjective (A = hypothesis):Conditional probability:Bayes' theorem.

5 14 Glen CowanMultivariate statistical methods in Particle PhysicsTest statisticsThe decision boundary is a surface in the n dimensional space ofinput variables, , y x = can treat the y(x) as a scalar test statistic or discriminatingfunction, and try to define this function so that its distribution has the maximum possible separation between the event types: y x f y H1 f y H0 ycutThe decision boundaryis now effectively a singlecut on y(x), dividing x space into tworegions: R0 (accept H0) R1 (reject H0)15 Glen CowanMultivariate statistical methods in Particle PhysicsClassification viewed as a statistical testProbability to reject H0 if it is true (type I error): = R1f y H0 dyProbability to accept H0 if H1 is true (type II error): = R0f y H1 dy = significance level, size of test, false discovery rate = power of test with respect to H1 Equivalently if H0 = background, H1 = signal, use efficiencies: s= R1f y H1 dy=1 =power b= R0f y H0 dy=1 16 Glen CowanMultivariate statistical methods in Particle PhysicsPurity / misclassification rateConsider the probability that an event assigned to a certain category is classified correctly ( , the purity).

6 Purity depends on the prior probabilities for an event to besignal or background (~s, b cross sections).posterior probabilityprior probabilityUse Bayes' theorem:Here R1 is signal region17 Glen CowanMultivariate statistical methods in Particle PhysicsROC curveWe can characterize the quality of a classification procedurewith the receiver operating characteristic (ROC curve)Independent of prior under ROC curve can be used as a measure of quality (but usuallyinterested in a specific or narrow range of efficiencies). efficiencybackground rejection= 1 background Glen CowanMultivariate statistical methods in Particle PhysicsConstructing a test statisticThe Neyman Pearson lemma states: to obtain the highest backgroundrejection for a given signal efficiency (highest power for a givensignificance level), choose the acceptance region for signal such thatp x s p x b cwhere c is a constant that determines the signal , the optimal discriminating function is given by thelikelihood ratio:y x =p x s p x b any monotonic function of this is just as Glen CowanMultivariate statistical methods in Particle PhysicsBayes optimal analysisFrom Bayes' theorem we can compute the posterior odds.

7 P s x p b x =p x s p x b p s p b which is proportional to the likelihood placing a cut on the likelihood ratio is equivalent to ensuringa minimum posterior odds ratio for the selected oddslikelihoodratioprior odds20 Glen CowanMultivariate statistical methods in Particle PhysicsPurity vs. efficiency trade offThe actual choice of signal efficiency (and thus purity) will dependon goal of analysis , ,Trigger selection (high efficiency)Event sample used for precision measurement (high purity)Measurement of signal cross section: maximizeDiscovery of signal: maximize expected significance ~s/ s bs/ b21 Glen CowanMultivariate statistical methods in Particle PhysicsNeyman Pearson doesn't always helpThe problem is that we usually don't have explicit formulae for the pdfs p(x|s), p(x|b), so for a given x we can't evaluate the likelihood we may have Monte Carlo models for signal and backgroundprocesses, so we can produce simulated data : x~p x s x1, , xNsgenerate x~p x b x1, , xNbgenerate training data events of known typeNaive try.

8 Enter each (s,b) event into an n dimensional histogram,use M bins for each of the n dimensions, total of Mn is potentially large prohibitively large number of cells to populate,can't generate enough training Glen CowanMultivariate statistical methods in Particle PhysicsStrategies for event classificationA compromise solution is to make an Ansatz for the form of thetest statistic y(x) with fewer parameters; determine them (usingMC) to give best discrimination between signal and , try to estimate the probability densities p(x|s), p(x|b)and use the estimated pdfs to compute the likelihood ratio at thedesired x values (for real data ).23 Glen CowanMultivariate statistical methods in Particle PhysicsAnsatz: Fisher: maximizeChoose the parameters w1, .., wn so that the pdfshave maximum separation.

9 We want: s byf (y)tblarge distance between mean values, small widthstsLinear test statisticy x = i=1nwixi= wT xf y s ,f y b J w = s b 2 s2 b224 Glen CowanMultivariate statistical methods in Particle PhysicsCoefficients for maximum separationFor the mean and variance of we find Vk ij= x k i x k jp x Hk d x k i= xip x Hk d xy x k= y x p x Hk d x= wT k k2= y x k sp x Hk d x= wTVk wWe havewhere k = 0, 1 (hypothesis)and i, j = 1, .., n (component of x)mean, covariance of x25 Glen CowanMultivariate statistical methods in Particle PhysicsThe numerator of J(w) is and the denominator is between classes within classes maximize 0 1 2= i,j=1nwiwj 0 1 i 0 1 j= i,j=1nwiwjBij= wTB w 02 12= i,j=1nwiwj V0 V1 ij= wTW wJ w = wTB w wTW w=separationbetweenclassesseparationwith inclassesDetermining the coefficients w26 Glen CowanMultivariate statistical methods in Particle PhysicsSettinggives Fisher s linear discriminant function:Fisher discriminant functionwith w W 1 0 1 y x = wT xH0H1 Gives linear decision of points in direction of decision boundary gives maximum separation.

10 J wi=027 Glen CowanMultivariate statistical methods in Particle PhysicsWe obtain equivalent separation between the classes if we multiply the wi by a common scale factor and add an offset w0:Thus we can fix the mean values t0 and t1 for the two classes to arbitrary values, , 0 and maximizingmeans minimizingMaximizing Fisher s J(w) least squares Comment on least squaresy x =w0 i=1nwixiJ w = 0 1 2/ 02 12 02 12=E0[ y 0 2] E1[ y 1 2]Ek[ y k 2] 1Nk i=1Nk yi k 2 Estimate expectation values with averages of training ( MC) data :28 Glen CowanMultivariate statistical methods in Particle PhysicsFisher discriminant for Gaussian dataSuppose f(x|Hk) is a multivariate Gaussian with mean valuesE0[ x]= 0forH0E1[ x]= 1forH1and covariance matrices V0 = V1 = V for both. We can write theFisher's discriminant function (with an offset) isy x =w0 0 1 V 1 xThe likelihood ratio is thusp x H0 p x H1 =exp[ 12 x 0 TV 1 12 x 1TV 1 x 1 ]=ey29 Glen CowanMultivariate statistical methods in Particle PhysicsFor non Gaussian data this no longer holds, but lineardiscriminant function may be simplest practical try to transform data so as to better approximateGaussian before constructing Fisher for Gaussian data (2)That is, y(x)