Example: bachelor of science

Boosted Partial Least-Squares Regression

'&$% Boosted PLS Regression : S eminaire F enelon 2008-1 Boosted Partial Least-SquaresRegressionJean-Fran cois DurandMontpellier II University, FranceE-Mail: site: '&$% Boosted PLS Regression : S eminaire F enelon Machine Learning versus Data Mining The data mining prediction process Partial Least-Squares Boosted by introduction to splines Few words on smoothing splines Regression splines Two sets of basis functions Bivariate Regression splines Least-Squares Splines Penalized PLS Regression What isL2 Boosting? Ordinary PLS viewed as aL2 Boost algorithm The linear PLS Regression : algorithm and model The building-model stage: choosing M PLS Splines (PLSS): a main effects additive model The PLSS model Choosing the tuning parameters Example 1: Multi-collinearity and outliers, the orange juice data Example 2: The Fisher iris data revisited by PLS boosting MAPLSS to capture interactions The ANOVA type model for main effects and interactions The

Boosted PLS Regression: S¶eminaire J.P. F¶enelon 2008-2I. Introduction { Machine Learning versus Data Mining { The data mining prediction process

Tags:

  Tesla, Square, Partial, Regression, Boosted partial least squares regression, Boosted

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Transcription of Boosted Partial Least-Squares Regression

1 '&$% Boosted PLS Regression : S eminaire F enelon 2008-1 Boosted Partial Least-SquaresRegressionJean-Fran cois DurandMontpellier II University, FranceE-Mail: site: '&$% Boosted PLS Regression : S eminaire F enelon Machine Learning versus Data Mining The data mining prediction process Partial Least-Squares Boosted by introduction to splines Few words on smoothing splines Regression splines Two sets of basis functions Bivariate Regression splines Least-Squares Splines Penalized PLS Regression What isL2 Boosting? Ordinary PLS viewed as aL2 Boost algorithm The linear PLS Regression : algorithm and model The building-model stage: choosing M PLS Splines (PLSS): a main effects additive model The PLSS model Choosing the tuning parameters Example 1: Multi-collinearity and outliers, the orange juice data Example 2: The Fisher iris data revisited by PLS boosting MAPLSS to capture interactions The ANOVA type model for main effects and interactions The building-model stage Example 3: Comparison between MAPLSS, MARS and BRUTO on simu-lated data Example 4: Multi-collinearity and bivariate interaction, the chem '&$% Boosted PLS Regression .

2 S eminaire F enelon Learning versus Data miningMachine Learning: machine learning is concerned with the designand development of algorithms and techniques that allow comput-ers to learn . The major focus of machine learning research isto extract information from data automatically, by computationaland statistical methods.[ ]Data mining: Data mining has been defined as the nontrivial ex-traction of implicit, previously unknown, and potentially useful in-formation from data and the science of extracting useful informa-tion from large data sets or databases. [ ]To be franc, I do not like very much the word Machine Learn-ing . I do prefer Data Mining that suggests helping humans tolearn rather than More concretely, many automaticblack-box methods involve thresholds for decision rules whose val-ues may be more or less well understood and controlled by the lazyuser who mostly accepts the default values proposed by the theR-package, called PLSS for Partial Least-Squares Splines,I tried to make the automatic part, inescapable to face with thehuge amount of data and computation, easy to master by on-lineconversational controls.

3 '&$% Boosted PLS Regression : S eminaire F enelon 2008-4 The data mining prediction processThe timing of the data mining prediction process to be followedwith PLSS functions, can be split up into 3 upthe aims of the problemand the associated building-model phase: a round-trip until obtaininga validated an evolutionary training data base following the re-tained schedule the Regression and validate or not the model builton the data at ascenario of prediction. A scenario allows the userto conveniently enter new real or fictive observations to test thevalidated Least-Squares Boosted by splinesPartial Least-Squares Regression [19, S.]

4 Wold et al.], in short PLS,may be viewed as a repeated Least-Squares fit of residuals fromregressions on latent variables that are linear compromises of thepredictors and of maximum covariance with the responses. Thismethod is presented here in the framework ofL2boosting methods[11, Hastie et al.], [8, Friedman], by considering PLS compo-nents as the base learners.'&$% Boosted PLS Regression : S eminaire F enelon 2008-5 Historically very popular in chemistry and now in many scien-tific domains, Partial Least-Squares Regression of responsesYonpredictorsX,PLS(X, Y), produces linear models and has been re-cently extended to ANOVA style decomposition models called PLSS plines, PLSS, and Multivariate Additive PLS Splines, MAPLSS,[4], [5], [12].

5 The key point of this nonlinear approach was inspired by thebook of A. Gifi[9]who replaced in exploratory data analysis meth-ods, the design matrixXby the super-coding matrixBfrom trans-forming the variables byB-splines. LetBibe the coding of pre-dictoribyB-splines, PLSS produces main effects additive modelsPLSS(X, Y) PLS(B, Y)whereB= [B1|..|Bp], while capturing main effects plus bivariateinteractions leads toMAPLSS(X, Y) PLS(B, Y)whereB= [B1|..|Bp ..|Bi,i |..],Bi,i being the tensor prod-uct of splines for the two predictorsiandi .The aim of this course is twofold, first to detail the theoryof that way of boosting PLS that involves Regression splines in thebase learner, second to present real and simulated examples treatedby the free PLSS package available.

6 '&$% Boosted PLS Regression : S eminaire F enelon introduction to splinesFew words on smoothing splinesConsider the signal plus noise modelyi=s(xi) + i, i= 1.. n, x1< .. < xn [0,1]( 1.. , n) N(0, 2In n), 2is unknown ands Wm2[0,1]Wm2[0,1] ={s/s(l)l= 1, .. , m 1 are absolutely continuous, ands(m) L2[0,1]}.The smoothing spline estimator ofsiss =argmin Wm2[0,1]1nn i=1(yi (xi))2+ 10[ (m)(t)]2dt; >0usuallym= 2, to penalize convexity and solutions is unique and belongs to the space ofunivariatenatural spline functionsof degree 2m 1 (usually 3) withknotsat distinct data pointsx1< .. < xn.'&$% Boosted PLS Regression : S eminaire F enelon 2008-7 Now is time to tell something about what splines are!

7 To transform a continuous variablexwhose values range within[a, b], a spline functionsis made of adjacent polynomials of degreedthat join end to end at points called the knots , with continuityconditions for the derivatives[1, De Boor].145 100 50050100xs(x)spline of degree 2 with knots (2,3) on [1,5]23 Two kinds of splines that mainly differ by the way they are used: Smoothing splines: the degree is fixed to 3 and knots are locatedat distinct data points, the tuning parameter is a positivenumber that controls the smoothness. Regression splines: few knots{ j}jwhose number ad locationconstitute, joined to the degree, the tuning parameters.

8 Splinesare computed according to a Regression model.'&$% Boosted PLS Regression : S eminaire F enelon 2008-8 Regression splinesA spline belongs toa functional linear spaceS(m,{ m+1, .. , m+K},[a, b])of dimensionm+Kcharacterized bythree tuning parameters-the degreedor the orderm=d+ 1 of the polynomials,-the numberKand-the location of knots{ m+1, .. , m+K} 1=..= m=a < m+1 .. m+K< b= m+K+1=..= 2m+ splines S(m,{ m+1, .. , m+K},[a, b]) can be writtens(x) =m+K i=1 iBmi(x)where{Bmi(.)}i=1,..,m+Kis a basis of spline vector of the coordinate values is to be estimated by aregression method.'&$% Boosted PLS Regression : S eminaire F enelon 2008-9 Two sets of basis functions The truncated power functionsx (x )d+1234502468xtruncated power functions at knot 2d=1d=3d=2d=0 When knots are distinct, a basis ofS(m,{ m+1.)}

9 , m+K},[a, b])is given by1, x, .. , xd,(x m+1)d+, .. ,(x m+K)d+Notice that, whenK= 0,S(m, ,[a, b]) = the set polynomials of ordermon [a, b].'&$% Boosted PLS Regression : S eminaire F enelon 2008-10 TheB-splinesB-splines of degreed(orderm=d+ 1): forj= 1, .. , m+K,Bmj(x) = ( 1)m( j+m j)[ j, .. , j+m](x )d+where [ j, .. , j+m](x )d+is the divided difference of ordermcomputed at j, .. , j+mfor the function (x )d+.This basis is the most popular partly due to the next propertythat allows to compute recursively the values ofB-splines,[1,De Boor]B1j(x) = 1 if j x j+1,0 otherwise,Fork= 2, .. , m,Bkj(x) =x j j+k 1 jBk 1j(x) + j+k x j+k j+1Bk 1j+1(x).

10 Many statistical packages implement those formulae to computethem+Kvalues of theB-splines given anxsampleR-package:library(splines)x=seq (1,2*pi,length=100)B=bs(x,degree=2,knots =c(pi/2,3*pi/2),intercept=T)# What are the dimensions of the matrix B?'&$% Boosted PLS Regression : S eminaire F enelon 2008-11 The attractiveB-splines family for coding (a) (b) (2, ,[0,5])(a),S(3, ,[0,5])(b) andS(1,{1,2,4},[0,5]).'&$% Boosted PLS Regression : S eminaire F enelon forS(2,{1,2,4},[0,5])andS(3,{1,2,4},[0,5 ]).'&$% Boosted PLS Regression : S eminaire F enelon 2008-13 Local support:Bmi(x) = 0, x / [ i, i+m]. One observationxi, has a local influence ons(xi) thatdepends only on thembasis functions whose supports encom-pass this data.


Related search queries