Gaussian Processes for Regression: A Quick Introduction

1 Speak , ,GPRisaless parametric ,it snotcompletelyfree-form,andifwe reunwillingtomakeevenbasicassumptionsabo ut,thenmoregen-eraltechniquesshouldbecon sidered,includingthoseunderpinnedbythepr incipleofmaximumentropy;Chapter6ofSiviaa ndSkilling(2006)offersanintroduction. 1 2 1 :Givensixnoisydatapoints(errorbarsareind icatedwithverticallines), (GPs) , ,theobservationsinanarbitrarydataset,,ca nalwaysbeimaginedasasinglepointsampledfr omsomemultivariate(-variate)Gaussiandist ri-bution, ,workingbackwards, ,it ,.Apopularchoiceisthe squaredexponential ,(1)wherethemaximumallowablecovarianceis definedas ,thenapproachesthismaximum, :forourfunctiontolooksmooth, ,wehaveinstead, see ,forexample,duringinterpolationatnewvalu es, ,,sothereismuchflexibilitybuiltinto(1).

2 Notquiteenoughflexibilitythough:thedataa reoftennoisyaswell, :(2)somethingwhichshouldlookfamiliartoth osewho ,wetakethenovelapproachoffoldingthenoise into,bywriting(3)whereistheKroneckerdelt afunction.(WhenmostpeopleuseGaussianproc esses, , ,givenobservations,ourobjectiveistopredi ct,notthe actual ;theirexpectedvaluesareidenticalaccordin gto(2), ,theexpectedvalueof,andof,isthedotat.)To prepareforGPR,wecalculatethecovariancefu nction,(3),amongallpossiblecombinationso fthesepoints,summarizingourfindingsinthr eematrices:..(4)(5)Confirmforyourselftha tthediagonalelementsofare, ,wehavethat(6) : giventhedata,howlikelyisacertainpredicti onfor? .AsexplainedmoreslowlyintheAppendix,thep robabilityfollowsaGaussiandistribution:( 7)Ourbestestimateforisthemeanofthisdistr ibution:(8)andtheuncertaintyinourestimat eiscapturedinitsvariance:(9)We , (moreonthislater),wehaveenoughtocalculat eacovariancematrixusing(4):From(5) (8)and(9), , ,asshowninFigure2.

3 (Infact,equivalently, ,sincethereare1,000testpointsspreadovert heaxis,wouldbeofsize1,0001,000.)Ratherth anplottingsimpleerrorbars,we vedecidedtoplot,givinga95% 1 2 1 :Thesolidlineindicatesanestimationoffor1 , callthem arenotcho-sensensibly, theoremtellsusthat,assumingwehavelittlep riorknowledgeaboutwhatshouldbe,thiscorre spondstomaximizing,givenby(10)Simplyruny ourfavouritemultivariateoptimizationalgo rithm( ,Nelder-Meadsimplex,etc.)onthisequationa ndyou vefoundaprettygoodchoicefor;inourexample , sonly prettygood because,ofcourse, ,whenyoucanintegrateeverythingovertheman ydifferentpossiblechoicesfor?Chapter5ofR asmussenandWilliams(2006) ,ifyoufeelyou vegraspedthetoyprobleminFigure2, (a),inadditiontoalong-termdownwardtrend, hassomefluctuations,sowemightuseamoresop histicatedcovariancefunction:(11)Thefirs ttermtakesintoaccountthesmallvicissitude softhedependentvariable,andthesecondterm hasalongerlengthparameter()torepresentit slong-term4 101234567 5 4 3 2 1012345xy 10123456 4 20246xy(a)(b)Figure3:Estimationof(solidl ine)forafunctionwith(a)short-termandlong -termdynamics,and(b) , ,butit sconsideranotherfunction,whichwe (b)wasregressedwiththefollowingcovarianc efunction.

4 (12)Thefirsttermrepresentsthehill-liketr endoverthelongterm, veencounteredacasewhereandcanbedistantan dyetstill see eachother(thatis,for).Whatifthedependent variablehasotherdynamicswhich,apriori,yo uexpecttoappear?There snolimittohowcomplicatedcanbe, (2006)offersagoodoutlineoftherangeofcova riancefunctionsyoushouldkeepinyourtoolki t. Hangonaminute, youask, isn tchoosingacovariancefunctionfromatoolkit alotlikechoosingamodeltype,suchaslinearv ersuscubic whichwediscussedattheoutset? Well, ,thereisnowaytoperformregressionwithouti mposingatleastamodicumofstructureontheda taset; ,it ,thereexistsexcellenttheoreti-caljustifi cationfortheuseof(1)inmanysettings(Rasmu ssenandWilliams(2006), ). , vepresentedabriefoutlineofthemathematics ofGPR, tagoodcomputerprogrammer,thenthecodeforF igures1and2 , vemerelyscratchedthesurfaceofapowerfulte chnique(MacKay,1998).

5 First,althoughthefocushasbeenonone-dimen sionalinputs,it , ,thezerovectorrepresentingthemeanofthemu ltivariateGaussiandistributionin(6) ,inadditiontotheiruseinregression,GPsare applicabletointegration,globaloptimizati on,mixture-of-expertsmodels,unsuper-vise dlearningmodels,andmore seeChapter9ofRasmussenandWilliams(2006). ,D.(1998). (Ed.),Neuralnetworksandmachinelearning.( NATOASIS eries,SeriesF,ComputerandSystemsSciences , , ) , (2006). , (2006).DataAnalysis:ABayesianTutorial(se conded.). inotherwords,writingwouldbethesameaswrit ing(13)where,, , ,thenknowingwouldn ttellusanythingabout:specifically,.Onthe otherhand,ifwerenonzero,thensomematrixal gebraleadsusto(14)Themean,,isknownasthe matrixofregressioncoefficients ,andthevari-ance,,isthe Schurcomplementofin.

6 Insummary,ifweknowsomeof,wecanusethattoi nformourestimateofwhattherestofmightbe, ,August2008 Prerequisitereading:GaussianProcessesfor Regression1 OVERVIEWA smentionedinthepreviousdocument, ,iftheoutputofaGPissquashedontotherange, itcanrepresenttheprobabilityofadatapoint belongingtooneofsaytwotypes,andvoil`a, ,,arelinkedtotheunderlyingfunctionoutput s,.Theyarenolongerconnectedsimplyviaanoi seprocessasin(2)inthepreviousdocument, ,wecouldtryfittingaGPthatproducesanoutpu tofapproximatelyforsomevaluesofandapprox imatelyforothers, ,weinterposetheGPbetweenthedataandasquas hingfunction;then, latentfunction , ,data,GPlatentfunction,sigmoidclassproba bility,. ,soperhapswe representingthingsinreverseorder!

7 , ,herewewillprescribeittobethecumulativeG aussiandistribution,.This-shapedfunction satisfiesourneeds,mappinghighinto, ,revisiting(6)and(7)inthefirstdocument:c onfirmforyour-selfthat,iftherewerenonois e(),thetwoequationscouldberewrittenas(1) and(2)12 USINGTHECLASSIFIERS upposewe vetrainedaclassifierfrominputdata,,andth eircorrespondingexpert-labelledoutputdat a,.Andsupposethatintheprocessweformedsom eGPoutputscorrespondingtothesedata, renowreadytoinputanewdatapoint,,inthelef tsideofourschematic, ,findingtheprobabilityissimilartoGPR, (2):(3)(willbeexplainedsoon,butfornowcon siderittobeverysimilarto.)Inthesecondste p,wesquashtofindtheprobabilityofclassmem bership,.Theexpectedvalueis(4)Thisisthei ntegralofacumulativeGaussiantimesaGaussi an, (2006),thesolutionis:(5) ,sothatweknoweverythingabouttheGPpro-duc ing(3), ,naturallywe ,howlikelytheyaretobeappropriateforthetr ainingdatacanbedecomposedusingBayes theorem:(6)Let ,(7)Droppingthesubscriptsintheproduct,is informedbyoursigmoidfunction.

8 Specifically,isbydefinition,andtocomplet ethepicture,. ,butfirstwe 20 1001020 10 8 6 4 20246810xLatent function f(x) 20 probabilityFigure1:(a)Toyclassificationd ataset,wherecirclesandcrossesindicatecla ssmem-bershipoftheinput(training)data, , (mean)probabilityprob, answer toourproblemaftersuccessfullyperformingG PC.(b)Thecorrespondingdistributionofthel atentfunction, (6)withrespecttoiszero,orequivalentlyand moresimply, ,andusingthesamelogicthatproduced(10)int hepreviousdocument,wefindthat(8) ,appearsonbothsidesoftheequation,sowemak eaninitialguess(zeroisfine) (8)canbeuseddirectlyin(3),sowe (6),whichturnsouttobe, ,wepretendisGaussiandistributed, (9)(Thisassumptionisoccasionallyinaccura te,soifityieldspoorclassifications,bet-t erwaysofcharacterizingtheuncertaintyinsh ouldbeconsidered,forexampleviaexpectatio npropagation.)

9 (2)directlyisin-appropriate:inparticular , ,(3), (2),,isbeingmultipliedby,weaddcovtotheva riancein(2).Simplificationleadsto(3), ,we , , ,asusual,isoptimizedbymaximizing,or(omit tingontherighthandsideoftheequation),(10 )Thiscanbesimplified,usingaLaplaceapprox imation,toyield(11)Thisistheequationtoru nyourfavouriteoptimizeron, vedescribedbinaryclassification,wherethe numberofpossibleclasses,, , ,ourGPvaluesareconcatenatedas(12)Letbeav ectorofthesamelengthaswhich,foreach, (2006) (merelyone-dimensional)cumulativeGaussia ndis-tributionisnolongersufficienttodesc ribethesquashingfunctioninourclassifier; ,(13)whereisanonconsecutivesubsetof, vepresentedthetwobigchangesneededtogofro mbinary-tomulti-classGPC, (6),wereplace(8)with(14)Thecorresponding varianceisasbefore,butnowdiag,whereisama trixobtainedbystackingverticallythediago nalmatricesdiag, ,wehaveenoughtogeneralize(3)todiag(15)wh ere,, ,(11)isreplacedwith(16)Wewon tpresentanexampleofmulti-classGPC, ,classificationcanbeextendedtoacceptvalu eswithmultipledimen-sions, ,puttingconfidenceintervalsontheclassifi cationprobabili-ties,calculatingthederiv ativesof(16)toaidtheoptimizer,orusingthe variationalGaussianprocessclassifiersdes cribedinMacKay(1998), , seeforexampleChapter5ofSiviaandSkilling( 2006)

10 ,we veagainsparedyouafewpracticalalgorithmic details; , ,aswellastheALADDIN project( ).REFERENCESMacKay,D.(1998). (Ed.),Neuralnetworksandmachinelearning.( NATOASIS eries,SeriesF,ComputerandSystemsSciences , , ) , (2006). , (2006).DataAnalysis:ABayesianTutorial(se conded.).