MATH 3P82 REGRESSION ANALYSIS Lecture Notes

1 MATH 3P82 REGRESSION ANALYSISL ecture Notesc Jan Vrbik23 Contents1 PREVIEW52 USINGMAPLE7 Basics .. 9 Procedures .. 10 MatrixAlgebra .. 10 Otherusefulcommands: .. 113 SIMPLE REGRESSION13 MaximumLikelihoodMethod .. 15 ConfidenceIntervals .. 17 REGRESSION coefficients .. 18 Residual 19 Hypotheses 20 ModelAdequacy(Lack-of-FitTest).. 20 Weighted REGRESSION .. 22 Correlation .. 24 Large -Sample 26 Confidence interval for the correlation 274 multivariate (LINEAR) REGRESSION29 MultivariateNormalDistribution .. 29 Partial correlation 30 Multiple REGRESSION - Main 33 Weighted-case 35 SearchingforOptimalModel .. 37 CoefficientofCorrelation(Determination) .. 384 Polynomial REGRESSION .. 39 Dummy(Indicator)Variables .. 425 NONLINEARREGRESSION436 ROBUSTREGRESSION47 Laplace distribution .. 47 CauchyCase .. 53 YuleModel.

2 555 Chapter 1 PREVIEWR egression is a procedure which selects, from a certain class of functions, the onewhich bestfits a given set of empirical data (usually presented as a table ofxandyvalues with, inevitably, some random component). The independent variablexisusually called theregressor(there may be one or more of these), the dependent variableyis The random components (calledresiduals)are usually assumednormallydistributed, with the same and independent of class from which the functions are selected (themodel) is usually one ofthe following types:1. a linear function ofx( +bx)- simple (univariate) linear REGRESSION ,2. a linear function ofx1,x2,..xk- multiple ( multivariate ) linear REGRESSION ,3. a polynomial function ofx- polynomial REGRESSION ,4. any other type of function, with one or more parameters ( )-nonlinear coefficients (parameters) of these models are calledregression coeffi-cients(parameters).

3 Our main task is going to be tofind goodestimatorsofthe REGRESSION coefficients (they should have correct expected values and variancesas small as possible), to be used forpredictingvalues ofywhen new observationsare of the related issues are:1. How do know (can we test) whether the relationship (betweenyandx)is truly linear? What if it is not (we have switch to either polynomial ornonlinear model).2. Similarly, are the residuals truly normal and independent of each other? Howdo wefix the procedure if the answer is Even when they are normal and independent, what if their variance changeswithx(here, we have to do the so calledweighted REGRESSION ).4. Even when all the assumptions are properly met: In the multivariate casewith many independent variables, do we really need them all to make a goodprediction abouty?And, if itispossible to reduce them (usually substan-tially) to a smaller subset, how do we do it ( selecting thebestfive,say).

4 ?67 Chapter 2 USING MAPLEB asicsTyping anexpression(following Maple s>prompt) results in evaluating the expression contains only integers (no decimal point), one gets the exact(rational) answer, as soon as at least one number in the expression is real (with adecimal point), the result is real (rounded offto 10 significant digits). The symbols ,/and facilitate multiplication, division and exponentiation, respectively. Notethat each line of your input has to end with a semicolon:>4 5 3/(5 + 2) + 2 ( 3) ;110356 The result of any computation can be stored under a name (which you make up,rather arbitrarily), and used in any subsequent expression. Maple then remembersthe value, until the end of your session, or till you deliberately replace it with anew value. Note that this (giving a name to a result) is achieved by typing thename, followed by a colon and the equal sign (a group of two symbols, representinga single operation), followed by the actual expression to be stored:>a:= ( +4) (2 6) + 2/3 4/5;a:= 333>a/7+9; 238>a:= 14/6;a:=73;>a/7+9;a:=283;(from now on, we will omit the>prompt from our examples, showing onlywhatwehave to type).

5 Maple can also handle the usual functions such assin, cos, tan, arcsin,arccos, arctan, exp, ln, sqrt, etc. All angles are always measured in (3.);sqrt(8) ;.14112 000812 2We can alsodefineour own functions by:f:=x >x 2;8f:=x x2f(3);9wherefis an arbitrary and LoopsMaple can store, under a single name, a wholelistof values, thus:a:= [3/2,5,sqrt(3),7] ;a:= [32,5, 3,7]The individual elements of a list can be referenced by indexing (and used incomputing another expression):a[2] 4;20 One can add elements of a list by the followingcommand(as Maple callsthem):sum( a[i] , i = ) ;272+ 3 One can convert the last answer to its decimal form by:evalf(%) ; 081 Note that the%symbol always refers to the previous tosum, one can also computeproductof elements of a subtract say3from each element of the lista,redefiningacorrespondingly,canbeac hievedby:forifrom1to4doa[i]:=a[i] 3end do:Note that terminating a statement by:instead of the usual.

6 Will preventMaple from printing the four results computed in the process (we may not needto see them individually). Also note that, upon completion of this statement,iwill have the value of5(any informationihad contained previously will have beendestroyed)!We can easily verify that the individual elements of ouralist have been updatedaccordingly:a[2] ;2We may also create a list using the following approach:b:= [seq(2 i, i= )] ;b:= [2,4,8,16,32,64 ] ;9 Variables and PolynomialsIf a symbol, such as for examplex,has not been assigned a specificvalue,Mapleconsiders it a variable. We may then defineato be apolynomialinx,thus:a:= 3 2 x+4 x 2 ;a:= 3 2x+4x2A polynomial can be differentiateddiff(a, x); 2+8xintegrated from, say,0to3int(a, x= ) ;36or plotted, for a certain range ofxvaluesplot(a, x= ) ;We can also evaluate it, substituting a specificnumberforx(there are actuallytwo ways of doing this):subs(x=3,a);eval(a, x=3);3333We can also multiply two polynomials (in our example, we will multiplyabyitself), but to convert to a regular polynomial form, we nee toexpandthe answer:a a;expand(%).

7 (3 2x+4x2)29 12x+28x2 16x3+16x410 ProceduresIf some specific computation (consisting, potentially, of several steps) is to be done,more than once ( we would like to be able to raise each element of a list ofvalues to a given power), we needfirst to design the correspondingprocedure(effectively a simple computer program), for example:RAISETO:=proc(L, N);localK, n, i;K:=L;n:=nops(L);forifrom1tondoK[i]:=K[ i] Nend do;Kend proc:whereRAISETOis an arbitrary name of the procedure,LandNare arbitrarynames of itsarguments(also called parameters), thefirst for the list and thesecond for the exponent,K,nandiare auxiliary names to be used in the actualcomputation (since they arelocal,theywillnotinterferewithanysuch namesused outside the procedure). First we copyLintoK(Maple does not like it if wetry to modifyLdirectly) andfind its lengthn(by thenopscommand).

8 Then,we raise each elementKto the power ofN,and return (the last expression of theprocedure) the modified list. We can organize the procedure into several lines byusing Shift-Enter (to move to the next line).We can then use the procedure as follows:SV FL([2,5,7,1],2);SV FL([3,8,4], 1) ;[4,25,49,1][13,18,14]Matrix AlgebraWe can define a matrix by:a:=matrix(2,2,[1,2,3,4]):where2,2spec ifies its dimensions (number of rows and columns, respectively),followed by the list of its elements (row-wise).We can multiply two matrices (here, we multiplyaby itself) byevalm(a& a):Note that we have to replace the usual by & .Similarly, we can add andsubtract (using+and ),and raiseato any positive integer power (using ).We can also multiplyaby a vector (of matching length), which can be enteredas a list:evalm(a& [2,5]):Note that reversing the order ofaand[2,5]yields a different can also compute thetransposeandinverseofa,butfirstwemust askMaple to make these commands available by:with(linalg):11We can then perform the required operation bytranspose(a) , to solve a set of linear equation withabeing the matrix of coefficientsand[2,3]the right hand side vector, we do:linsolve(a,[2,3]):Other useful commands:a:=randmatrix(5,5):creates a matrix of specified dimensions with random elements,augument(a,[6,2,7,1,0]):attache s the list, making it an extra (last) column ofa,submatrix(a, , ):reducesato a3by2submatrix, keeping only rows2,3and4,and columns1and2,swaprow(a,2,5):interchanges rows 2 and 5 ofa,addrow(a,2,4,2/3).

9 Adds row2multiplied by23to recall the proper syntax of a command, one can always type:?addrowto get its whole-page description, usually with a specific function (or several functions) is easy (as we have already seen):plot({sin(x),x x 3/6},x= ):One can also plot a scattergram of individual points (it isfirst necessary to askMaple to make to corresponding routine available, as follows:with(plots):pointplot( [[0,2],[1, 3],[3,0],[4,1],[7, 2]]);Note that the argument was alistof pairs ofx-yvalues (each pair itself enclosedin brackets).We can combine any two such plots (usually a scattergram of points togetherwith afitted polynomial) by:pic1:=pointplot([seq([i/5,sin(i/5)],i = )] ):pic2:=plot(sin(x),x= ):display(pic1,pic2):1213 Chapter 3 SIMPLE REGRESSIONThe model isyi= 0+ 1xi+ i( )wherei=1,2, .., n,making the following assumptions:1. The values ofxare measured exactly , with no random error.)

10 This is usuallyso when we can choose them at The iare normally distributed, independent of each other (uncorrelated),having the expected value of0and variance equal to 2(thesameforeachofthem, regardless of the value ofxi).Note that the actual value of is usuallynot two REGRESSION coefficients are called theslope and values are also unknown, and need to be estimated using the empirical dataat suchestimators,weusetheMaximum Likelihood Methodwhich is almost always the best tool for this kind of task. It guarantees to yieldestimators which areasymptotically unbiased, having the smallest possiblevariance. It works as follows:1. We write down the joint probability density function of theyi s (note thatthesearerandom variables).2. Considering it a function of the parameters ( 0, 1and in this case)only( freezing theyi s at theirobservedvalues), wemaximizeit, using theusual techniques.