Chapter 14 Factor analysis - York University

1 ,Y2,:::,Yl,arelinearlyrelatedtoasmallern umberofunob-servablefactorsF1,F2,:::, ,however,thatundercertainconditionsthehy pothesizedfactormodelhascertainimplicati ons, ,andhowthemodelcanbetested, nance, ,Y2,andY3,respectively,representastudent ' vestudents(ina10-pointnumericalscaleabov ethepassingmark), ,Y1 Marketing,Y2 Policy,Y31365273331098439751065c PeterTryfos, :FactoranalysisIthasbeensuggestedthatthe segradesarefunctionsoftwounderlyingfacto rs,F1andF2,tentativelyandratherlooselyde scribedasquantitativeabilityandverbalabi lity, ,asfollows:Y1= 10+ 11F1+ 12F2+e1Y2= 20+ 21F1+ 22F2+e2Y3= 30+ 31F1+ 32F2+e3(14:1)Theerrortermse1,e2,ande3, ,theparameters , , nanceishighlyquantitative, nance, ,itisexpectedthattheloadingshaveroughlyt hefollowingstructure:Loadingon:Variable, YiF1, i1F2, i2Y1+0Y20+Y30+Thegradeinthe nancecourseisexpectedtobepositivelyrelat edtoquantitativeabilitybutunrelatedtover balability;thegradesinmarketingandpolicy ,ontheotherhand, , `0'wemeanapproximatelyequaltozeroandby`+ 'apositivenumbersubstantiallydi ,however, ,Y2andY3accordingto( ).

2 Thesimplestmodeloffactoranalysisisbasedo ntwoassumptionsconcerningtherelationship s( ).Weshall :Theerrortermseiareindependentofoneanoth er,andsuchthatE(ei)=0andVar(ei)= ( ).A2:TheunobservablefactorsFjareindepend entofoneanotherandoftheerrorterms,andare suchthatE(Fj)=0andVar(Fj)= , , , ;sincethefactorsarenotobservable, ,andcanbewrittenasYi= i0+ i1F1+ i2F2+(1) :Var(Yi)= 2i1 Var(F1)+ 2i2 Var(F2)+(1)2 Var(ei)= 2i1+ 2i2+ 2i:WeseethatthevarianceofYiconsistsoftwo parts:Var(Yi)= 2i1+ 2i2|{z}communality+ 2i|{z}speci cvariance:The rst,thecommunalityofthevariable, ,thespeci cvariance, ,thene1=e2=e3=0always,and 21= 22= 23=0:Tocalculatethecovarianceofanytwoobs ervablevariables,YiandYj,wecanwriteYi= i0+ i1F1+ i2F2+(1)ei+(0)ej;andYj= j0+ j1F1+ j2F2+(0)ei+(1)ej; nd4 Chapter14:FactoranalysisCov(Yi.)

3 Yj)= i1 j1 Var(F1)+ i2 j2 Var(F2)+(1)(0)Var(ei)+(0)(1)Var(ej)= i1 j1+ i2 j2:Wecanarrangeallthevariancesandcovaria ncesintheformofthefollowingtable:Variabl e:Variable:Y1Y2Y3Y1 211+ 212+ 21 21 11+ 22 12 31 11+ 32 12Y2 11 21+ 12 22 221+ 222+ 22 21 31+ 22 32Y3 11 31+ 12 32 21 31+ 22 32 231+ 232+ 23 WehaveplacedthevariancesoftheYvariablesi nthediagonalcellsofthetableandthecovaria nceso ' ( ).Thematrixissymmetric,inthesensethatthe entryinrow1andcolumn2isthesameasthatinro w2andcolumn1, ,Y2,andY3,wecancalculatetheobservedvaria ncesandcovariancesofthevariables,andarra ngethemintheobservedvariancecovariancema -trix:Variable:Variable:Y1Y2Y3Y1S21S12S1 3Y2S21S22S23Y3S31S32S23 Thus,S21istheobservedvarianceofY1,S12the observedcovarianceofY1andY2, ,ofcourse,theS12=S21,S13=S31,andsoon;the matrix,inotherwords, :0@9:84 0:360:44 0:365:043:840:443:843 ,therefore,wehavetheobservedvariancesand covari-ancesofthevariables.

4 Ontheother, 'sassumptionsaretrue,weshouldbeabletoest imatetheloadings ,but ,thatis,thatthereexistanin nitenumberofsetsofvaluesofthe rstanarbitrarysetofloadingsde ningwhatweshallcallModelA:Y1=0:5F1+0:5F2 +e1Y2=0:3F1+0:3F2+e2Y3=0:5F1 0:5F2+e3 Itisnotdi culttoverifythatthetheoreticalvarianceco variancematrixis0@0:5+ 210:300:30:18+ 220000:5+ 231 AForexample,Var(Y1)=(0:5)2+(0:5)2+ 21=0:5+ 21;Cov(Y1;Y2)=(0:5)(0:3)+(0:5)(0:3)=0:3; ,considerModelB,havingadi erentsetof ij:Y1=(p2=2)F1+0F2+e1Y2=(0:3p2)F1+0F2+e2 Y3=0F1 (p2=2)F2+e3 Itcanagainbeeasilycon ,Var(Y1)=(p2=2)2+(0)2+ 21=0:5+ 21;Cov(Y1;Y2)=(p2=2)(0:3p2)+(0)(0)=0:3; (a) cientofF1,andalongtheverticalaxisthecoe cientsofF1andF2inthe rstequationareplottedasthepointwithcoord inates( , );thoseofthesecondequationasthepoint( , ),andthoseofthethirdasthepoint( , 0:5).

5 6 Chapter14 (b) , (c),thepointwithoriginalcoordinates( , )nowhascoordinates(0:3p2,0). (d). ,ofcourse,in , ,itisexpectedthatsomeloadingswillbeclose tozero,whileotherswillbepositiveornegati veandsubstantiallydi , rst,onesetofloadings ijiscalculatedwhichyieldstheoreticalvari ancesandcovariancesthat ,how-ever,maynotagreewiththepriorexpecta tions, ,inthesecondstage,the rstload-ingsare\rotated"inane orttoarriveatanothersetthat tequallywelltheobservedvariancesandcovar iances, ,forexample,thattheloadingsofModelAhappe nedtobetheestimatesproducingthebest , ,however, ttheobservedvariancesandcovariancesequal lywellandindicateclearlythatY1andY2depen dononefactoronly, rstsetofloadingsistheprincipalcomponentm ethod.* (fortheexampleofthischapter)theelementso fthefactor* (alsocalledprincipalaxis) ,forexample,JohnsonandWichern(1992, ),Rencher(1995, ).

6 8 Chapter14 ,ObservedCommunality,Yivariance,S2i 2i1+ 2i2 Financegrade,Y1S21 211+ 212 Marketinggrade,Y2S22 221+ 222 Policygrade,Y3S23 231+ ,itwillberecalled, , ijwhichmakethetotalcommunality( )approximateascloselyaspossiblethesumoft heobservedvariancesofthevariables( ). ( ) * , ,unstandardizedvariablesVariable,Observe dLoadingsonCommunality,PercentYivariance ,S2iF1;bi1F2;bi2b2i1+b2i2explained(1)(2) (3)(4)(5)(6)=100 (5)/(2)Finance, , , ,tobedistiguishedfromthetheoreticalloadi ngs *BasedontheoutputofprogramSASwiththestat ementsprocfactorn=2covvardef=neigenvecto rs; (twoorthreedecimalplaceswouldhavebeensu cient); ; ,F1couldbeinterpretedasquantitative,andF 2asverbal, , , nance, , ,Pib2i1,canbeinterpretedasthecontributio nofF1,andthatonF2,Pib2i2, ,Xib2i1=(3:136773)2+ +(0:127697)2=9:873125.

7 AndXib2i2=(0:023799)2+ +(1:731884)2=8:007997:Thus, , cvarianceofYi, 2i,isthedi erencebetweentheobservedvarianceandestim atedcommunalityofYi:Variable, , , , (ine ect,attachingthesameweighttoeachvariable )makessensewhentheYvariablesaremeasuredi nthesameunits, ,however, , (Yij)themeanofthevariable( Yi)anddividingtheresultbythestandarddevi ation(Si)ofthevariabletoobtainthestandar dizedobservationY0ij,Y0ij=Yij YiSi:10 Chapter14:FactoranalysisItcanbeshownthat thecovariancesofthestandardizedvariables areequaltothecorrelationcoe cientsoftheoriginalvariables(thevariance softhestandardizedvariablesare,ofcourse, equalto1).Thislastresultcanbeeasilyveri ,wecalculateY1Y2Y3 Mean, ,S2 , :Y01Y02Y03 ,the rststandardizedvariableisgivenbyY01j=Y1j 6:63:1369:Itcanbecon rmedthatthemeansofthestandardizedvariabl esareequalto0, Y1 Y2=(216) (6:6)(6:6)= 0:36:Thecorrelationcoe cientofY1andY2isr12=S12S1S2= 0:36(3:1369)(2:245)= 0:0511;andisequaltothecovarianceofY01and Y02,S012=1nXY01Y02 Y01 Y02=( 0:2556) (0)(0)= 0:0511:Ingeneral,Cov(Y0i;Y0j)=Cor(Yi;Yj) .

8 Thecorrelationmatrixoftheoriginalvariabl es(equaltothecovariancematrixofthestanda rdizedvariables)is 1 0:05110:0804 0:051110:98100:08040:98101!Thereaderisas kedtocon ,ine ect,subjectstheobservedcorrelationmatrix oftheoriginalvariables|ratherthantheobse rvedvariancecovariancematrix| ( ,SPSS,SAS), * , ,standardizedvariablesStandardizedObserv edLoadingsonCommunality,Percentvariable, Y0ivariance,S02iF1;bi1F2;bi2b2i1+b2i2exp lained(1)(2)(3)(4)(5)(6)=100 (5)/(2)Finance, , , ,marketingandpolicygradesdependononecomm onfactor(whichcanbeinterpretedasver-bala bility)butnotappreciablyontheother(quant itativeability);there-verseholdsfor ,however, , (thelabelsF1andF2are,ofcourse,arbitrary) . ,slightlylessthanwiththeoriginalvariable s.

9 *Readersfamiliarwithlinearalgebramaywant toknowthattheprinci-palcomponentsolution involvestheeigenvalues(characteristicval ues)andeigenvectors(characteristicvector s) ,forexample,JohnsonandWichern, , rstfactorsolutiondoesnotrevealthehypothe sizedstructureoftheloadings,itiscustomar ytoapplyrotationinane ortto ndanothersetofloadingsthat , ;thegoalistomakesomeoftheseloadingsaslar geaspossible, ,ontheotherhand,seekstomaximizethevarian ceofthesquaredloadingsforeachvariable, , ,instructedtoapplythevarimaxrotationtoth e ,butthecontributionsofeachfactordi ,rotationdidnotalterappreciablythe , ,standardizedvariablesStandardizedObserv edLoadingsonCommunality,Percentvariable, Y0ivariance,S02iF1;bi1F2;bi2b2i1+b2i2exp lained(1)(2)(3)(4)(5)(6)=100 (5)/(2)Finance, , , , ,however, ,unlessinstructedotherwise,onlyidentify( \extract") , ,two,three, ,andtotailorthehypothesisto ,however,somesubjectivityindeclaringload ingstobe\high"or\closetozero" rstandsubsequentfactorsolutions,andeachc ombinationof rstsolutionandrotationmethodmaygiveriset oentirelydi , ,Percentvariable,Y0ivariance,S02iF1;bi1F 2;bi2b2i1+b2i2explained(1)(2)(3)(4)(5)(6 )=100 (5)/(2) :AREA:Floorarea,insquarefeetLOTSZ:Sizeof thelot,insquarefeetROOMS: rstfewpropertiesinthe , , ,isitnottruethatlargehousestendtobebuilt onlargelots,andtohavelarge oorarea,moreroomsandmorebathrooms?

10 ,theprogramusedstandardizedvariables,the built-incriterionfordeterminingthenumber offactors,andtheprincipalcomponentmethod ; :FactoranalysisContrarytopriorexpectatio ns,twofactors|notone|wereidenti rstfactorhashighloadingsforAREA,ROOMS,an dBATHS; , ,therefore, , Factoranalysisisamethodforinvestigatingw hetheranumberofvariablesofinterestarelin earlyrelatedtoasmallernumberofunobservab lefactors. Inthespecialvocabularyoffactoranalysis,t heparametersoftheselinearfunctionsareref erredtoasloadings. Undercertainconditions(A1andA2inthetext) ,thetheoreticalvarianceofeachvariableand thecovarianceofeachpairofvariablescanbee xpressedintermsoftheloadingsandthevarian ceoftheerrorterms. cvarianceisthepartofthevarianceofthevari ablethatisnotaccountedbythecommonfactors .