Transcription of Model Selection: General Techniques - Stanford University
1 - p. 1/16 Statistics203:IntroductiontoRegressionan dAnalysisofVarianceModelSelection:Genera lTechniquesJonathanTaylorlTodaylCrudeout lierdetectiontestlBonferronicorrectionlS imultaneousinferencefor lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 2/16 TodaynOutlierdetection/ (Some) lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 3/16 CrudeoutlierdetectiontestnIf thestudentizedresidualsarelarge:observat ionmay :ifnis large, if we threshold att1 =2;n p 1wewillgetmany outliersby chanceevenif modelis :Bonferronicorrection,thresholdatt1 =2n;n p lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.
2 4/16 BonferronicorrectionnIf we aredoingmanyt(orother)tests, saym >1we cancontroloverallfalsepositive rateat by testingeachoneatlevel = :P(at leastonefalsepositive)=P [mi=1jTij t1 =2m;n p 1 mXi=1P jTij t1 =2m;n p 1 =mXi=1 m= :nKnownas simultaneousinference :controllingoverallfalsepositive rateat whileperformingmany lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 5/16 Simultaneousinferencefor nOthercommonsituationsin whichsimultaneousinferenceoccursis simultaneousinference for .nUsingthefactsthatb N ; 2(XtX) 1 b 2 2 2n pn palongwithb ?b 2leadsto( b )t(XtX)(b )=pb 2 2p=p 2n p=(n p) Fp;n pn(1 ) 100%simultaneousconfidenceregion:n : ( b )t(XtX)(b ) pb 2Fp;n p;1 olTodaylCrudeoutlierdetectiontestlBonfer ronicorrectionlSimultaneousinferencefor lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.]
3 6/16 Modelselection:goalsnWhenwe have many predictors(withmany possibleinteractions),it canbedifficultto finda include?nWhichinteractionsdowe include?nModelselectiontriesto simplify lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 7/16 Modelselection:generalnThisis an unsolved problemin statistics:therearenomagicproceduresto getyouthe bestmodel. nIn somesense, modelselectionis datamining. nDataminers/ machinelearnersoftenwork withvery lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.
4 8/16 Modelselection:strategiesnTo implement this, we need:ua criterionorbenchmark tocomparetwo limitednumberof predictors, it is possible tosearchallpossible lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 9/16 PossiblecriterianR2: nota > optimum is to take : better. It penalized s InformationCriterion(AIC),Schwarz s lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 10/16 Mallow sCpnCp(M) =SSE(M)b 2 n+ 2 p(M):nb 2=SSE(F)=dfFis the best estimateof 2we have (usethefullestmodel)nSSE(M) =kY bYMk2is theSSEof themodelMnp(M)is thenumberofpredictorsinM, orthedegreesoffreedomusedupby 2nXi=1E (Yi E(Yi))2 =1 2nXi=1E (Yi bYi)2 +Var(bYi)lTodaylCrudeoutlierdetectiontes tlBonferronicorrectionlSimultaneousinfer encefor lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.
5 11/16 AIC& BICnMallow sCpis (almost)a specialcaseofAkaike InformationCriterion(AIC)AIC(M) = 2 logL(M) + 2 p(M):nL(M)is thelikelihoodfunctionof theparametersin modelMevaluatedattheMLE(MaximumLikelihoo dEstimators).nSchwarz s BayesianInformationCriterion(BIC)BIC(M) = 2 logL(M) +p(M) lognlTodaylCrudeoutlierdetectiontestlBon ferronicorrectionlSimultaneousinferencef or lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 12/16 MaximumlikelihoodestimationnIf themodelis correctthenthelog-likelihoodof( ; )islogL( ; jX; Y) = n2 log(2 ) + log 2 12 2kY X k2whereYis in thiscaseis thesameasleastsquaresestimatebecausefirs tterm doesnotdependon nMLEfor 2:@@ 2logL( ; ) b ;b 2= n2 2+12 4kY Xb k2= 0nSolvingfor 2:b 2M LE=1nkY Xb k2=1nSSE(M)NotethattheMLEis lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.
6 13/16 AICfora linearmodelnUsingb M LE=b b 2M LE=1nSSE(M)we seethattheAICofa multiplelinearregressionmodelisAIC(M) =n(log(2 ) + log(SSE(M)) log(n))+2(n+p(M)+1)nIf 2is known,thenAIC(M) =n log(2 ) + log( 2) +SSE(M) 2+ 2p(M)whichis almostCp(M) + lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 14/16 Search strategiesn Bestsubset :searchallpossible modelsandtake (forward,backwardorboth):usefulwhenthenu mberofpredictorsis large. Chooseaninitialmodelandbe greedy .n Greedy meansalwaystake thebiggestjump(upordown)in lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p.
7 15/16 ImplementationsinRn Bestsubset :usethefunctionleaps. :usethefunctionstep. Worksforany modelwithAkaike InformationCriterion(AIC).Inmultipleline arregression,AICis (almost)a lModelselection:goalslModelselection:gen erallModelselection:strategieslPossible criterialMallow sCplAIC& BIClMaximumlikelihoodestimationlAICfora linearmodellSearchstrategieslImplementat ionsinRlCaveats- p. 16/16 CaveatsnMany other criteria have wellforsometypesof data, arenot directmeasures of predictive we willseecross-validationwhichis anestimateofpredictive power.
