Example: quiz answers

Classification and Regression by randomForest

,December200218 ClassificationandRegressionbyrandomFores tAndyLiawandMatthewWienerIntroductionRec entlytherehasbeenalotofinterestin ensem-blelearning (see, ,Shapireetal.,1998)andbaggingBreiman(199 6) , , ,successivetreesdonotdependonearliertree s , (2001)proposedrandomforests, , , , ,includingdiscriminantanalysis,supportve ctorma-chinesandneuralnetworks,andisrobu stagainstoverfitting(Breiman,2001).Inadd ition,itisveryuser-friendlyinthesensetha tithasonlytwoparam-eters(thenumberofvari ablesintherandomsubsetateachnodeandthenu mberoftreesintheforest), ( ). (forbothclassificationandregression) ,growanun-prunedclassificationorregressi ontree,withthefollowingmodification:atea chnode,ratherthanchoosingthebestsplitamo ngallpredic-tors,randomlysamplemtryofthe predictorsandchoosethebestsplitfromamong thosevariables.

structed, random forest, with the default m try, we were able to clearly identify the only two informa-tive variables and totally ignore the other 998 noise variables. A regression example We use the Boston Housing data (available in the MASSpackage)asanexampleforregressionbyran-dom forest. Note a few differences between classifi-

Fullscreen Download

Tags:

  Forest, Random, Random forests, Randomforest

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Advertisement

Transcription of Classification and Regression by randomForest

1 ,December200218 ClassificationandRegressionbyrandomFores tAndyLiawandMatthewWienerIntroductionRec entlytherehasbeenalotofinterestin ensem-blelearning (see, ,Shapireetal.,1998)andbaggingBreiman(199 6) , , ,successivetreesdonotdependonearliertree s , (2001)proposedrandomforests, , , , ,includingdiscriminantanalysis,supportve ctorma-chinesandneuralnetworks,andisrobu stagainstoverfitting(Breiman,2001).Inadd ition,itisveryuser-friendlyinthesensetha tithasonlytwoparam-eters(thenumberofvari ablesintherandomsubsetateachnodeandthenu mberoftreesintheforest), ( ). (forbothclassificationandregression) ,growanun-prunedclassificationorregressi ontree,withthefollowingmodification:atea chnode,ratherthanchoosingthebestsplitamo ngallpredic-tors,randomlysamplemtryofthe predictorsandchoosethebestsplitfromamong thosevariables.

2 (Baggingcanbethoughtofasthespecialcaseof randomforestsobtainedwhenmtry=p,thenumbe rofpredictors.) ( ,majorityvotesforclassification,averagef orregression).Anestimateoftheerrorrateca nbeobtained,basedonthetrainingdata, ,predictthedatanotinthebootstrapsample(w hatBreimancalls out-of-bag ,orOOB,data) (Ontheav-erage,eachdatapointwouldbeout-o f-bagaround36%ofthetimes,soaggregatethes epredictions.)Calcuatetheerrorrate, ,giventhatenoughtreeshavebeengrown(other wisetheOOBestimatecanbiasupward;seeBylan der(2002)).ExtrainformationfromRandomFor estsTherandomForestpackageoptionallyprod ucestwoadditionalpiecesofinformation:ame asureoftheimportanceofthepredictorvariab les,andameasureoftheinternalstructureoft hedata(theproximityofdifferentdatapoints tooneanother).

3 VariableimportanceThisisadifficultconcep ttodefineingeneral,becausetheimportanceo favariablemaybeduetoits(possiblycomplex) (OOB) ( (2002)fortheirdefinitions.)proximitymeas ureThe(i,j) similar ,December200219toidentifystructureinthed ata(seeBreiman,2002)orforunsupervisedlea rningwithran-domforests(seebelow).Usagei nRTheuserinterfacetorandomforestisconsis tentwiththatofotherclassificationfunctio ns suchasnnet()(inthennetpackage)andsvm()(i nthee1071pack-age).(Weactuallyborrowedso meoftheinterfacecodefromthosetwofunction s.)Thereisaformulainterface,andpredictor scanbespecifiedasamatrixordataframeviath exargument, ,randomForestperformsclassification;ifth eresponseiscontinuous(thatis,notafactor) , ,randomForestperformsunsupervisedlearnin g(seebelow).

4 (orelsetheywillbewronglytreatedascontinu ous).TherandomForestfunctionreturnsanobj ectofclass" randomForest ". (VenablesandRipley,2002) :>library( randomForest )>library(MASS)>da ta(fgl)> (17)> <- randomForest (type~.,data=fgl,+mtry=2,importance=TRUE,+ )100: OOB error rate= : OOB error rate= : OOB error rate= : OOB error rate= : OOB error rate= >print( ) (formula = type ~ .,data = fgl, mtry = 2, importance = TRUE, = 100)Type of random forest : classificationNumber of trees: 500No. of variables tried at each split: 2 OOB estimate of error rate: matrix:WinF WinNF Veh Con Tabl Head 63 6 1 0 0 0 9 62 1 2 2 0 7 4 6 0 0 0 0 2 0 10 0 1 0 2 0 0 7 0 1 2 0 1 0 25 ,usingtheerrorestfunctionsintheipredpack age:>library(ipred)> (131)> <-numeric(10)>for(iin1:10) [i]<-+errorest(type~.)

5 ,data=fgl,+model= randomForest ,mtry=2)$er ror>summary( )Min. 1st Qu. Median Mean 3rd Qu. >library(e1071)> (563)> <-numeric(10)>for(iin1:10) [i]<-+errorest(type~.,data=fgl,+ model = svm, cost = 10, gamma = )$error>summary( )Min. 1st Qu. Median Mean 3rd Qu. ( ,usethe important variablestobuildsimpler,morereadilyinter pretablemodels).Figure1showsthevariablei mportanceoftheForensicGlassdataset, ,itiscreatedby>par(mfrow=c(2,2))>for(iin 1:4)+ plot(sort( $importance[,i], dec = TRUE),+type="h",main=paste("Measure",i)) ,K,andFefromthemodel,theerrorratere-main sbelow20%.

6 ,December200220 Measure 1 RIMgCaBaSiAlFeKNa010203040 Measure 2 RIMgAlCaBaKNaSiFe051015 Measure 4 AlMgRICaNaKSiBaFe0510152025 Figure1 ,weusedthevariableimportancemeasurestose lectonlydozensofpredictors, ,000variablesthatwecon-structed,randomfo rest,withthedefaultmtry, (availableintheMASS package) : Thedefaultmtryisp/3,asopposedtop1/2forcl assification,wherepisthenumberofpredic-t ors. Thedefaultnodesizeis5,asopposedto1forcla ssification.(Inthetreebuildingalgorithm, nodeswithfewerthannodesizeobservationsar enotsplitted.) Thereisonlyonemeasureofvariableimpor-tan ce,insteadoffour.>data(Boston)> (1341)> <- randomForest (medv ~.)

7 , Boston)>print( ) (formula = medv ~ .,data = Boston)Type of random forest : regressionNumber of trees: 500No. of variables tried at each split: 4 Mean of squared residuals: meanofsquaredresiduals iscomputedasMSEOOB=n 1n 1{yi yOOBi}2,where percentvarianceex-plained iscomputedas1 MSEOOB 2y,where 2yiscomputedwithnasdivisor(ratherthann 1).Wecancomparetheresultwiththeactualdat a,aswellasfittedvaluesfromalinearmodel, Plot MatrixRF101020203030303040405050LM001010 2020202030304040 Actual101020203030303040405050 Figure2 , trick istocallthedata class1 andconstructa class2 syntheticdata, class2 class2 dataaresampledfromtheprod-uctofthemargin aldistributionsofthevari-ables(byindepen dentbootstrapofeachvari-ableseparately).

8 , class2 dataaresampleduniformlyfromthehypercubec ontainingthedata(bysam-plinguniformlywit hintherangeofeachvari-ables).Theideaisth atrealdatapointsthat aresimilartooneanotherwillfrequentlyendu pinthesameter-minalnodeofatree exactlywhatismeasuredbytheproximitymatri xthatcanbereturnedusingtheproximity= , 309ofMASS4(alsofoundinlines28 29and63 68in $RHOME/library/MASS/ ),result-ingthe ()(inpackagemva)tovisualizethe1 proximity, ,thetwocolorformsarefairlywellseparated. >library(mva)> (131)> <- randomForest (dslcrabs,+ ntree = 1000, proximity = TRUE)$proximity> <-cmdscale( )>plot( ,col=c("blue",+"orange")[codes(crabs$sp) ],pch=c(1,+ 16)[codes(crabs$sex)], xlab="", ylab="") B/MB/FO/MO/FFigure3 ,which,ifsettoTRUE,returnsamea-sureof outlyingness (assumingtheyareunlabelled).

9 Somenotesforpracticaluse ,youhaveenoughtrees. Forselectingmtry, ,halfofthedefault,andtwicethedefault, , !Ifonehasaverylargenumberofvariablesbute xpectsonlyveryfewtobe important ,usinglargermtrymaygivebetterperformance . ,ourexperiencehasbeenthateventhoughtheva riableimportancemeasuresmayvaryfromrunto run,therankingoftheimpor-tancesisquitest able. Forclassificationproblemswheretheclassfr e-quenciesareextremelyunbalanced( ,99%class1and1%class2), ,inatwo-classprob-lemwith99%class1and1%c lass2,onemaywanttopredictthe1%oftheobser vationswithlargestclass2probabilitiesasc lass2,andusethesmallestofthoseprobabilit iesasthresh-oldforpredictionoftestdata( ,usethetype= prob argumentinthepredictmethodandthresholdth esecondcolumnoftheout-put).

10 Bydefault, , ,onlyonetreeiskeptinmemoryatanytime, ,December200222memory(andpotentiallyexec utiontime)canbesaved. Sincethealgorithmfallsintothe embarrass-inglyparallel category, , ,andpoint-ingoutthereferenceBylander(200 2). ,24(2):123 140, ,45(1):5 32, ,using, , , ,48:287 297, , , , , ,26(5):1651

Show more

Related documents

Using Random Forest to Learn Imbalanced Data

Using Random Forest to Learn Imbalanced Data

statistics.berkeley.edu

2.1 Random Forest Random forest (Breiman, 2001) is an ensemble of unpruned classification or regression trees, induced from bootstrap samples of the training data, using random feature selection in the tree induction process. Predic-tion is made by aggregating (majority vote for classification or averaging for regression) the predictions of

  Forest, Random, Random forests, Random forest random forest

1 RANDOM FORESTS - University of California, Berkeley

1 RANDOM FORESTS - University of California, Berkeley

www.stat.berkeley.edu

number of independent random integers between 1 and K. The nature and dimensionality of Θ depends on its use in tree construction. After a large number of trees is generated, they vote for the most popular class. We call these procedures random forests. Definition 1.1 A random forest is a classifier consisting of a collection of tree-

  Forest, Random, Random forests

Package ‘randomForest’

Package ‘randomForest’

cran.r-project.org

Title Breiman and Cutler's Random Forests for Classification and Regression Version 4.6-14 Date 2018-03-22 Depends R (>= 3.2.2), stats Suggests RColorBrewer, MASS Author Fortran original by Leo Breiman and Adele Cutler, R port by Andy Liaw and Matthew Wiener. Description Classification and regression based on a forest of trees using random in-

  Forest, Random, Randomforest

Title stata.com meta forestplot — Forest plots

Title stata.com meta forestplot — Forest plots

www.stata.com

meta forestplot— Forest plots 3 Syntax meta forestplot column list if in, options column list is a list of column names given by col. In the Meta-Analysis Control Panel, the columns can be specified on the Forest plot tab of the Forest plot pane. options Description Main random (remethod) random-effects meta-analysis common (cefemethod)

  Forest, Random

Fixed-Effect Versus Random-Effects Models

Fixed-Effect Versus Random-Effects Models

www.meta-analysis.com

In Chapter 11 and Chapter 12 we introduced the fixed-effect and random-effects models. Here, we highlight the conceptual and practical differences between them. Consider the forest plots in Figures 13.1 and 13.2. They include the same six studies, but the first uses a fixed-effect analysis and the second a random-effects analysis.

  Forest, Random

Methods for estimating above&#173;ground biomass of forest and ...

Methods for estimating above­ground biomass of forest and ...

ctfs.si.edu

The key to measuring above-ground biomass is to select locations at random and then measure all trees, alive or dead, and weigh ground-cover where …

  Forest, Random

Random Forests - Springer

Random Forests - Springer

link.springer.com

a number of independent random integers between 1 and K. The nature and dimensionality of depends on its use in tree construction. After a large number of trees is generated, they vote for the most popular class. We call these procedures random forests. Definition 1.1. A random forest is a classifier consisting of a collection of tree-structured

  Forest, Random, Random forests

Random Forest - Mathematics and Statistics

Random Forest - Mathematics and Statistics

www.math.mcgill.ca

Random Forest Gradient Boosting (5 Node) FIGURE 15.1. Bagging, random forest, and gradient boosting, applied to the spam data. For boosting, 5-node trees were used, and the number of trees were chosen by 10-fold cross-validation (2500 trees). Each “step” in the figure corre-sponds to a change in a single misclassification (in a test set ...

  Forest, Random, Random forests

Random Forest - univ-toulouse.fr

Random Forest - univ-toulouse.fr

perso.math.univ-toulouse.fr

Random forest > Random decision tree • All labeled samples initially assigned to root node • N ← root node • With node N do • Find the feature F among a random subset of features + threshold value T...

  Forest, Random, Random forests

Related search queries

Random forest, Random Forest Random forest, Random, RandomForest, Forest