Singular Value Decomposition & Independent …

1 HST582 ,2005 IntroductionInthischapterwewillexamineho wwecangeneralizetheideaoftransforminga timeseriesinanalternativerepresentation, suchastheFourier(frequency)domain,tofaci li-tatesystematicmethodsofeitherremoving ( ltering)oradding(interpolating) , wewillexaminethetechniquesofPrincipalCom ponentAnalysis(PCA)usingSingularValueDec omposition(SVD),andIndependentComponentA nalysis(ICA).Bothofthesetechniquesutiliz ea representationofthedataina statisticaldomainratherthana ,thedatais projectedontoa newsetofaxesthatful llsomestatisticalcriterion,whichimplyind ependence,ratherthana thattheFouriercomponentsontowhicha datasegmentis projectedare xed, If thestructureofthedatachangesovertime,the ntheaxesontowhichthedatais essentiallya methodforseparatingthedataoutintoseparat esourceswhichwillhopefullyallowustoseeim portantstructureina ,bycalculatingthepowerspectrumofa segmentofdata, (amplitudesquared)alongcertainfrequencyv ectorsis thereforehigh,meaningwehavea strongcomponentinthesignal1atthatfrequen cy.

2 Bydiscardingtheprojectionsthatcorrespond totheunwantedsources(suchasthenoiseorart ifactsources)andinvertingthetransformati on,weeffectivelyper-forma trueforbothICA andPCA , oneimportantdifferencebetweenthesetechni quesis andICA weattemptto nda assumetherearea setofindependentsourcesin thedata,butdonotassumetheirexactproperti es.(Therefore,theymayoverlapinthefrequen cydomainincontrasttoFouriertechniques.)W e thende , ratherthande nethetheaxes,thisprocessis knownasblindsourcesepara-tion. themeasureweusetodiscovertheaxesisvarian ceandleadstoa setoforthogonalaxes(becausethedatais decorrelatedin a secondordersenseandthedotproductof anyof theaxesis zero).ForICA thismeasureis basedonnon-Gaussianity(suchaskurtosis1- see ) thefourthmoment(mean,variance,andskewnes sarethe rstthree)andis a measureofhownon-Gaussiana thatif wemaximizethenon-Gaussianityofa setofsignals,thentheyaremaximallyindepen dent.(Thiscomesfromthecentrallimittheore m;if wekeepaddingindependentsignalstogether, wewilleventuallyarriveata Gaussiandistribution.)

3 If webreaka Gaussian-likeobservationdownintoa setof non-Gaussianmixtures,eachwithdis-tributi onsthatareasnon-Gaussianaspossible, ,kurtosisallowsustoseparatenon-Gaussiani ndependentsources, ,if formulatedinthecorrectmanner, canleadtosomesurprisingresults,asyouwill discoverintheapplicationssectionlaterint hesenotesandintheaccompa-nyinglaboratory . However, weshall noiseseparationIngeneral,anobserved(reco rded) datareductionstep( ltering)followedbya datareconstructiontechnique(suchasinterp olation).However, thesuccessofthedatareductionandreconstru ctionstepsis nition,noiseis thepartoftheobservationthatmaskstheunder lyingsignalwewish1a measureofhowwiderornarrowera distributionis thana Gaussian2toanalyze2, , fora noisesignaltocarrynoinformation,it mustbewhitewitha atspectrumandanautocorrelationfunction(A CF)equaltoanimpulse3. Mostrealnoiseis notreallywhite, ,thetermnoiseisoftenusedratherlooselyand is ,muscularactivityrecordedontheelectrocar diogram(ECG)is , increasedmuscleartifactontheECGactuallyt ellsusthatthesubjectis moreactivethanwhenlittleornomusclenoisei s thereforea sourceofinformationaboutactivity, althoughit nitionsarethereforetask-relatedandchange dependingonthenatureof illustratestherangeofsignalcontaminantsf ortheECG4.

4 We shallalsoexam-inethestatisticalqualities ofthesecontaminantsintermsoftheirprobabi litydistributionfunctions(PDFs)sincethep owerspectrumofa signalis notalwayssuf cienttocharac-terizea PDFcanbedescribedintermsofitsGaussianity , orrather,departuresfromthisidealizedform (whicharethereforecalledsuper-orsub-Gaus sian).ThefactthatthesesignalsarenotGauss ianturnsouttobeanextremelyimportantqual- ity, whichis closelyconnectedtotheconceptofindependen ce,whichweshallexploittoseparatecontamin antsformthesignalAlthoughnoiseis oftenmodeledasGaussianwhitenoise5, thisis oftencorrelated(withitselforsometimesthe signal), ,50 Hzor60 Hzmainsnoisecontaminationis sinusoidal,a waveformthatspendsmostofitstimeattheextr emevalues(nearitsturningpoints).Byconsid eringdeparturesfromtheidealGaussiannoise modelwewillseehowconventionaltechniquesc anunder-performandhowmoresophisticated(s tatistical-based)techniquescanprovideimp roved willnowexplorehowthisissimplyanotherform ofdatareduction(or ltering)throughprojectionontoa lterthedata(bydiscardingtheprojectionson toaxesthatarebelievedtocorrespondtonoise ).

5 Byprojectingfroma reducedsetofbasisfunctions(ontowhichthed atahasbeencompressed)backtotheoriginalsp ace,weperforma typeofinterpolation(byaddinginformationf roma modelthatencodessomeofourpriorbeliefsabo uttheunderlyingnatureofthesignaloris derivedfroma dataset).2it lowerstheSNR!3 Therefore,noone-steppredictionis descriptiveexamplebecauseit haseasilyrecognizable(andde nable) 3 Qualities FrequencyTimeContaminant Rangeduration50or60 HZPowerlineNarrowband50 Continuousor HzMovementBaselineNarrowbandTransientorW ander( Hz)ContinuousMuscleNoiseBroadbandTransie ntElectricalInterferenceNarrowbandTransi entorContinuousElectrodepopNarrowbandTra nsientObservationnoiseBroadbandContinuou sQuantizationnoiseBroadbandContinuousTab le1 :10secondsof3 rsttwosecondsandthe (S) ltersThesimplest lteringofa timeseriesinvolvesthetransformationofa discreteonedimen-sional( ) timeseries , consistingof pointssuchthat ,intoa newrepresentation,!" # %$& '$( ) '$( *$( +.))

6 If , -. / 0 1 2 is a columnvec-tor6thatrepresentsa channelofECG,thenwecangeneralizethisrepr esentationsothat channelsofECG3, andtheirtransformedrepresentation4aregiv enby35 67778 9 + 9 % : : : 9 <; + : : : =;.. > ? @ : : :A B;CEDDDF 4G 67778$& + $& % : : :$ $( $( + : : :$( =;..$/ > H$/ @ : : :I$/ B;CEDDDF(1)Notethatwewilladopttheconvent ionthroughoutthischapter(andintheaccompa nyinglaboratoryexercises)thatallvectorsa rewrittenin lower-caseboldandarecolumnvectors, pointsofeachofthe signalchannelsform JK matrices( -dimensionalwith samplesforeachvector).An( JL ) transformationmatrixMcanthenbeappliedto3 tocreatethetransformedmatrix4suchthat4 KNK OM 3PN (2)Thepurposeofa transformationis tomap(orproject) lterthedatawediscardthenoise,or`unintere sting'partsofthesignal(whicharemaskingth einformationweareinterestedin).Thisamoun tstoa dimensionalityreduction,aswearediscardin gthedimensions(orsubspace) , ,thetransformedsignalis samelength( ) astheoriginalandtheenergyofthedatais theDiscreteFouriertrans-form(DFT)whereth esamesignalis measuredalonga newsetofperpendicularaxescorrespondingto thecoef cientsoftheFourierseries(seechapter4).)) )

7 InthecaseoftheDFTwithQR S frequencyvectors, as4 UTV XW;Y Z M TY3 YwhereM TY O[]\]^ _`TY a;, orequivalentlyM 67778[\]^ _[\]^cb_: : :[\]^ _ ;[]\]^=b_[]\]^ed_: : :[]\]^=b_ ;..[]\]^ _` []\]^=b_ : : :X[]\]^ _ B;CDDDF (3)Forbiorthogonaltransforms,theanglesbe tweentheaxesmaychangeandthenewaxesarenot necessarilyperpendicular. However, noinformationis lostandperfectreconstructionoftheorigina lsignalis stillpossible(using3 OM\gf4 ).6 Inh jilk nmthecommandophUq`r`s]t uwv`x <y givesa dimensionofz#{}|anda lengthequalto~ (sothatthetransformationcanbereversedand theoriginaldatarestoredexactly)oraslossy . Whena signalis lteredorcompressed(throughdownsamplingfo rinstance),informationis oftenlostandthetransformationis ,lossytransformationsinvolvea non-invertibletransformationofthedatausi nga transformationmatrixthathasatleastonecol umnsetto anirreversibleremovalof someof thedataandthiscorrespondstoa mappingtoa (PCA)andIndependentComponentAnalysis(ICA ).

8 Bothtechniquesattemptto (ormapped)ontoeachvectoris , ICA resultsina andICA canbeusedtoperformlossyorlosslesstransfo rmationsbymultiplyingtherecorded(observa tion)databya andICAbothinvolveprojectingthedataontoa setofaxeswhicharedeterminedbythenatureof thedata,andarethereforemethodsofblindsou rceseparation(BSS).(Blindbecausetheaxeso fprojectionandthereforethesourcesaredete rminedthroughtheapplicationofaninternalm easureandwithouttheuseofanypriorknowledg eofthedatastructure.)However, oncewehavediscoveredthebasisfunctionsoft heindependentaxesinthedataandhaveseparat edthemoutbyprojectingthedataontotheseaxe s,wecanthenusethesetechniquesto andICA separationmatricesthatcorrespondtounwant edsourcestozero,weproducenoninvertiblema trices7. If wethenforcetheinversionoftheseparationma trix8andtransformthedatabackintotheorigi nalobservationspace, illustratestheBSSparadigmfor lteringwherebywehave (byalteringtheinverseofthedemixingmatrix tohavecolumnsofzerosfortheunwantedsource s), shallalsoseehowthesourcesthatwediscoverw ithPCA havea speci corderingaccordingtotheenergyalongeachax isfora becausewelookfortheaxisalongwhichthedata hasmaximumvariance(andhenceenergyorpower 9).

9 If thesignaltonoiseratio(SNR)is greaterthanunity, thesignalofinterestis thereforecon nedtothe , ICA allowsustodiscoversourcesbymeasuring7 Forexample,a transformationmatrix[10;0 0]isnoninvertible, orsingular(u o r so rinh nik nm) andmultiplyinga twodimensionalsignalbythismatrixperforms a pseudo-inversiontechniquesuchash nik nm t u ; u o r sOo 6a relativecostfunctionbetweenthesourcestha tis thereforenorelevancetotheorderofthecolum nsintheseparateddataandoftenwehavetoappl yfurthersignal-speci cmeasures,orheuristics, ,thebasisfunctionsforthenewrepresentatio nareprede nedandassumedtobeindependent,whereaswith PCA therepresentation, orthebasisfunctions, arefoundinthedatabylookingfora ,thedataundergoesa secondordersenseandareorthogonal(thedotp roductoftheaxes,andthecross-correlationo ftheprojectionsis closetozero).Thebasicideaintheapplicatio nofPCA toa dataset,is to ndthecomponentvectors! ,! ,..,!;thattheyexplainthemaximumamountofv ariancepossibleby canbede nedin anintuitivewayusinga rstprincipalcomponent is foundbypassingoverthedataandattemptingto maximizethevalueof Z 0 3.

10 Where is thesamelength asthedata3. Thusthe rstprincipalcomponentis theprojectiononthedirec-tioninwhichtheva rianceoftheprojectionis principalcomponentsarefoundbyrepeatingth isprocessin theremainingorthogonalsub-space(whichred ucesin dimensionalitybyoneforeachnewcomponentwe discover).Theprincipalcomponentsaretheng ivenby! 3( / w ), is todecorrelatethesignalbyperforminganorth ogonalprojection,weoftenreducethedimensi onofthedatafrom to ( ) canbeshownthatthePCA representationis anop-timallineardimensionreductiontechni queinthemean-squaresense[1].Oneimportant applicationofthistechniqueis fornoisereduction,wherethedatacontainedi nthelast componentsis tofthistechniqueis thata projectionintoa subspaceofa verylowdimension,forexampletwoorthree, ,thecomputationofthe canbesimplyaccomplishedusingthesampleco- variancematrix! ?3 3. The " aretheeigenvectorsof!(an J matrix)thatcorrespondtothe eigenvaluesof!. A methodfordeterminingtheeigenvaluesinthis manneris knownasSingularValueDecomposition(SVD),w hichis describedbelow.