Classification and Regression by randomForest

1 ,December200218 ClassificationandRegressionbyrandomFores tAndyLiawandMatthewWienerIntroductionRec entlytherehasbeenalotofinterestin ensem-blelearning (see, ,Shapireetal.,1998)andbaggingBreiman(199 6) , , ,successivetreesdonotdependonearliertree s , (2001)proposedrandomforests, , , , ,includingdiscriminantanalysis,supportve ctorma-chinesandneuralnetworks,andisrobu stagainstoverfitting(Breiman,2001).Inadd ition,itisveryuser-friendlyinthesensetha tithasonlytwoparam-eters(thenumberofvari ablesintherandomsubsetateachnodeandthenu mberoftreesintheforest), ( ). (forbothclassificationandregression) ,growanun-prunedclassificationorregressi ontree,withthefollowingmodification:atea chnode,ratherthanchoosingthebestsplitamo ngallpredic-tors,randomlysamplemtryofthe predictorsandchoosethebestsplitfromamong thosevariables.(Baggingcanbethoughtofast hespecialcaseofrandomforestsobtainedwhen mtry=p,thenumberofpredictors.) ( ,majorityvotesforclassification,averagef orregression).Anestimateoftheerrorrateca nbeobtained,basedonthetrainingdata, ,predictthedatanotinthebootstrapsample(w hatBreimancalls out-of-bag ,orOOB,data) (Ontheav-erage,eachdatapointwouldbeout-o f-bagaround36%ofthetimes,soaggregatethes epredictions.)

2 Calcuatetheerrorrate, ,giventhatenoughtreeshavebeengrown(other wisetheOOBestimatecanbiasupward;seeBylan der(2002)).ExtrainformationfromRandomFor estsTherandomForestpackageoptionallyprod ucestwoadditionalpiecesofinformation:ame asureoftheimportanceofthepredictorvariab les,andameasureoftheinternalstructureoft hedata(theproximityofdifferentdatapoints tooneanother).VariableimportanceThisisad ifficultconcepttodefineingeneral,because theimportanceofavariablemaybeduetoits(po ssiblycomplex) (OOB) ( (2002)fortheirdefinitions.)proximitymeas ureThe(i,j) similar ,December200219toidentifystructureinthed ata(seeBreiman,2002)orforunsupervisedlea rningwithran-domforests(seebelow).Usagei nRTheuserinterfacetorandomforestisconsis tentwiththatofotherclassificationfunctio ns suchasnnet()(inthennetpackage)andsvm()(i nthee1071pack-age).(Weactuallyborrowedso meoftheinterfacecodefromthosetwofunction s.)Thereisaformulainterface,andpredictor scanbespecifiedasamatrixordataframeviath exargument, ,randomForestperformsclassification;ifth eresponseiscontinuous(thatis,notafactor) , ,randomForestperformsunsupervisedlearnin g(seebelow).

3 (orelsetheywillbewronglytreatedascontinu ous).TherandomForestfunctionreturnsanobj ectofclass" randomForest ". (VenablesandRipley,2002) :>library( randomForest )>library(MASS)>da ta(fgl)> (17)> <- randomForest (type~.,data=fgl,+mtry=2,importance=TRUE,+ )100: OOB error rate= : OOB error rate= : OOB error rate= : OOB error rate= : OOB error rate= >print( ) (formula = type ~ .,data = fgl, mtry = 2, importance = TRUE, = 100)Type of random forest: classificationNumber of trees: 500No. of variables tried at each split: 2 OOB estimate of error rate: matrix:WinF WinNF Veh Con Tabl Head 63 6 1 0 0 0 9 62 1 2 2 0 7 4 6 0 0 0 0 2 0 10 0 1 0 2 0 0 7 0 1 2 0 1 0 25 ,usingtheerrorestfunctionsintheipredpack age:>library(ipred)> (131)> <-numeric(10)>for(iin1:10) [i]<-+errorest(type~.,data=fgl,+model= randomForest ,mtry=2)$error>summary( )Min. 1st Qu.

4 Median Mean 3rd Qu. >library(e1071)> (563)> <-numeric(10)>for(iin1:10) [i]<-+errorest(type~.,data=fgl,+ model = svm, cost = 10, gamma = )$error>summary( )Min. 1st Qu. Median Mean 3rd Qu. ( ,usethe important variablestobuildsimpler,morereadilyinter pretablemodels).Figure1showsthevariablei mportanceoftheForensicGlassdataset, ,itiscreatedby>par(mfrow=c(2,2))>for(iin 1:4)+ plot(sort( $importance[,i], dec = TRUE),+type="h",main=paste("Measure",i)) ,K,andFefromthemodel,theerrorratere-main sbelow20%. ,December200220 Measure 1 RIMgCaBaSiAlFeKNa010203040 Measure 2 RIMgAlCaBaKNaSiFe051015 Measure 4 AlMgRICaNaKSiBaFe0510152025 Figure1 ,weusedthevariableimportancemeasurestose lectonlydozensofpredictors, ,000variablesthatwecon-structed,randomfo rest,withthedefaultmtry, (availableintheMASS package) : Thedefaultmtryisp/3,asopposedtop1/2forcl assification,wherepisthenumberofpredic-t ors. Thedefaultnodesizeis5,asopposedto1forcla ssification.

5 (Inthetreebuildingalgorithm,nodeswithfew erthannodesizeobservationsarenotsplitted .) Thereisonlyonemeasureofvariableimpor-tan ce,insteadoffour.>data(Boston)> (1341)> <- randomForest (medv ~ ., Boston)>print( ) (formula = medv ~ .,data = Boston)Type of random forest: regressionNumber of trees: 500No. of variables tried at each split: 4 Mean of squared residuals: meanofsquaredresiduals iscomputedasMSEOOB=n 1n 1{yi yOOBi}2,where percentvarianceex-plained iscomputedas1 MSEOOB 2y,where 2yiscomputedwithnasdivisor(ratherthann 1).Wecancomparetheresultwiththeactualdat a,aswellasfittedvaluesfromalinearmodel, Plot MatrixRF101020203030303040405050LM001010 2020202030304040 Actual101020203030303040405050 Figure2 , trick istocallthedata class1 andconstructa class2 syntheticdata, class2 class2 dataaresampledfromtheprod-uctofthemargin aldistributionsofthevari-ables(byindepen dentbootstrapofeachvari-ableseparately). , class2 dataaresampleduniformlyfromthehypercubec ontainingthedata(bysam-plinguniformlywit hintherangeofeachvari-ables).

6 Theideaisthatrealdatapointsthat aresimilartooneanotherwillfrequentlyendu pinthesameter-minalnodeofatree exactlywhatismeasuredbytheproximitymatri xthatcanbereturnedusingtheproximity= , 309ofMASS4(alsofoundinlines28 29and63 68in $RHOME/library/MASS/ ),result-ingthe ()(inpackagemva)tovisualizethe1 proximity, ,thetwocolorformsarefairlywellseparated. >library(mva)> (131)> <- randomForest (dslcrabs,+ ntree = 1000, proximity = TRUE)$proximity> <-cmdscale( )>plot( ,col=c("blue",+"orange")[codes(crabs$sp) ],pch=c(1,+ 16)[codes(crabs$sex)], xlab="", ylab="") B/MB/FO/MO/FFigure3 ,which,ifsettoTRUE,returnsamea-sureof outlyingness (assumingtheyareunlabelled).Somenotesfor practicaluse ,youhaveenoughtrees. Forselectingmtry, ,halfofthedefault,andtwicethedefault, , !Ifonehasaverylargenumberofvariablesbute xpectsonlyveryfewtobe important ,usinglargermtrymaygivebetterperformance .

7 ,ourexperiencehasbeenthateventhoughtheva riableimportancemeasuresmayvaryfromrunto run,therankingoftheimpor-tancesisquitest able. Forclassificationproblemswheretheclassfr e-quenciesareextremelyunbalanced( ,99%class1and1%class2), ,inatwo-classprob-lemwith99%class1and1%c lass2,onemaywanttopredictthe1%oftheobser vationswithlargestclass2probabilitiesasc lass2,andusethesmallestofthoseprobabilit iesasthresh-oldforpredictionoftestdata( ,usethetype= prob argumentinthepredictmethodandthresholdth esecondcolumnoftheout-put). Bydefault, , ,onlyonetreeiskeptinmemoryatanytime, ,December200222memory(andpotentiallyexec utiontime)canbesaved. Sincethealgorithmfallsintothe embarrass-inglyparallel category, , ,andpoint-ingoutthereferenceBylander(200 2). ,24(2):123 140, ,45(1):5 32, ,using, , , ,48:287 297, , , , , ,26(5):1651