Bagging and Boosting - mit.edu

← Back to document page

Bagging and Class 10, 13 March 2006Sasha RakhlinPlan Bagging and sub-sampling methods bias - variance and stability for Bagging Boosting and correlations of machines Gradient descent view of boostingBagging (Bootstrap AGGregatING)Given a training setD={(x1, y1), . . .(xn, yn)}, sampleTsets ofnelements fromD(with replacement)D1, D2, . . . DT Tquasi replica training sets; train a machine on eachDi,i= 1, ..., Tand obtain asequence ofToutputsf1(x), . . . fT(x). Bagging (cont.)The final aggregate classifier can be for regression f(x) =T i=1fi(x),the average offifori= 1, ..., T; for classification f(x) = sign(T i=1fi(x))or the majority vote f(x) = sign(T i=1sign(fi(x)))Variation I: Sub-sampling methods- Standard Bagging : each of theTsubsamples has sizenand created with Sub- Bagging : createTsubsamples of size only ( <n).

Plan † Bagging and sub-sampling methods † Bias-Variance and stability for bagging † Boosting and correlations of machines † Gradient descent view of boosting

  Variance, Bias