Gaussian Processes for Regression: A Quick Introduction

1 Speak , ,GPRisaless parametric ,it snotcompletelyfree-form,andifwe reunwillingtomakeevenbasicassumptionsabo ut,thenmoregen-eraltechniquesshouldbecon sidered,includingthoseunderpinnedbythepr incipleofmaximumentropy;Chapter6ofSiviaa ndSkilling(2006)offersanintroduction. 1 2 1 :Givensixnoisydatapoints(errorbarsareind icatedwithverticallines), (GPs) , ,theobservationsinanarbitrarydataset,,ca nalwaysbeimaginedasasinglepointsampledfr omsomemultivariate(-variate)Gaussiandist ri-bution, ,workingbackwards, ,it ,.Apopularchoiceisthe squaredexponential ,(1)wherethemaximumallowablecovarianceis definedas ,thenapproachesthismaximum, :forourfunctiontolooksmooth, ,wehaveinstead, see ,forexample,duringinterpolationatnewvalu es, ,,sothereismuchflexibilitybuiltinto(1).

2 Notquiteenoughflexibilitythough:thedataa reoftennoisyaswell, :(2)somethingwhichshouldlookfamiliartoth osewho ,wetakethenovelapproachoffoldingthenoise into,bywriting(3)whereistheKroneckerdelt afunction.(WhenmostpeopleuseGaussianproc esses, , ,givenobservations,ourobjectiveistopredi ct,notthe actual ;theirexpectedvaluesareidenticalaccordin gto(2), ,theexpectedvalueof,andof,isthedotat.)To prepareforGPR,wecalculatethecovariancefu nction,(3),amongallpossiblecombinationso fthesepoints,summarizingourfindingsinthr eematrices:..(4)(5)Confirmforyourselftha tthediagonalelementsofare, ,wehavethat(6) : giventhedata,howlikelyisacertainpredicti onfor?

3 AsexplainedmoreslowlyintheAppendix,thepr obabilityfollowsaGaussiandistribution:(7 )Ourbestestimateforisthemeanofthisdistri bution:(8)andtheuncertaintyinourestimate iscapturedinitsvariance:(9)We , (moreonthislater),wehaveenoughtocalculat eacovariancematrixusing(4):From(5) (8)and(9), , ,asshowninFigure2.(Infact,equivalently, ,sincethereare1,000testpointsspreadovert heaxis,wouldbeofsize1,0001,000.)Ratherth anplottingsimpleerrorbars,we vedecidedtoplot,givinga95% 1 2 1 :Thesolidlineindicatesanestimationoffor1 , callthem arenotcho-sensensibly, theoremtellsusthat,assumingwehavelittlep riorknowledgeaboutwhatshouldbe,thiscorre spondstomaximizing,givenby(10)Simplyruny ourfavouritemultivariateoptimizationalgo rithm( ,Nelder-Meadsimplex,etc.)

4 Onthisequationandyou vefoundaprettygoodchoicefor;inourexample , sonly prettygood because,ofcourse, ,whenyoucanintegrateeverythingovertheman ydifferentpossiblechoicesfor?Chapter5ofR asmussenandWilliams(2006) ,ifyoufeelyou vegraspedthetoyprobleminFigure2, (a),inadditiontoalong-termdownwardtrend, hassomefluctuations,sowemightuseamoresop histicatedcovariancefunction:(11)Thefirs ttermtakesintoaccountthesmallvicissitude softhedependentvariable,andthesecondterm hasalongerlengthparameter()torepresentit slong-term4 101234567 5 4 3 2 1012345xy 10123456 4 20246xy(a)(b)Figure3.

5 Estimationof(solidline)forafunctionwith( a)short-termandlong-termdynamics,and(b) , ,butit sconsideranotherfunction,whichwe (b)wasregressedwiththefollowingcovarianc efunction:(12)Thefirsttermrepresentstheh ill-liketrendoverthelongterm, veencounteredacasewhereandcanbedistantan dyetstill see eachother(thatis,for).Whatifthedependent variablehasotherdynamicswhich,apriori,yo uexpecttoappear?There snolimittohowcomplicatedcanbe, (2006)offersagoodoutlineoftherangeofcova riancefunctionsyoushouldkeepinyourtoolki t. Hangonaminute, youask, isn tchoosingacovariancefunctionfromatoolkit alotlikechoosingamodeltype,suchaslinearv ersuscubic whichwediscussedattheoutset?

6 Well, ,thereisnowaytoperformregressionwithouti mposingatleastamodicumofstructureontheda taset; ,it ,thereexistsexcellenttheoreti-caljustifi cationfortheuseof(1)inmanysettings(Rasmu ssenandWilliams(2006), ). , vepresentedabriefoutlineofthemathematics ofGPR, tagoodcomputerprogrammer,thenthecodeforF igures1and2 , vemerelyscratchedthesurfaceofapowerfulte chnique(MacKay,1998).First,althoughthefo cushasbeenonone-dimensionalinputs,it , ,thezerovectorrepresentingthemeanofthemu ltivariateGaussiandistributionin(6) ,inadditiontotheiruseinregression,GPsare applicabletointegration,globaloptimizati on,mixture-of-expertsmodels,unsuper-vise dlearningmodels,andmore seeChapter9ofRasmussenandWilliams(2006).

7 ,D.(1998). (Ed.),Neuralnetworksandmachinelearning.( NATOASIS eries,SeriesF,ComputerandSystemsSciences , , ) , (2006). , (2006).DataAnalysis:ABayesianTutorial(se conded.). inotherwords,writingwouldbethesameaswrit ing(13)where,, , ,thenknowingwouldn ttellusanythingabout:specifically,.Onthe otherhand,ifwerenonzero,thensomematrixal gebraleadsusto(14)Themean,,isknownasthe matrixofregressioncoefficients ,andthevari-ance,,isthe Schurcomplementofin .Insummary,ifweknowsomeof,wecanusethatto informourestimateofwhattherestofmightbe, ,August2008 Prerequisitereading:GaussianProcessesfor Regression1 OVERVIEWA smentionedinthepreviousdocument, ,iftheoutputofaGPissquashedontotherange, itcanrepresenttheprobabilityofadatapoint belongingtooneofsaytwotypes,andvoil`a, ,,arelinkedtotheunderlyingfunctionoutput s.

8 Theyarenolongerconnectedsimplyviaanoisep rocessasin(2)inthepreviousdocument, ,wecouldtryfittingaGPthatproducesanoutpu tofapproximatelyforsomevaluesofandapprox imatelyforothers, ,weinterposetheGPbetweenthedataandasquas hingfunction;then, latentfunction , ,data,GPlatentfunction,sigmoidclassproba bility,. ,soperhapswe representingthingsinreverseorder! , ,herewewillprescribeittobethecumulativeG aussiandistribution,.This-shapedfunction satisfiesourneeds,mappinghighinto, ,revisiting(6)and(7)inthefirstdocument:c onfirmforyour-selfthat,iftherewerenonois e(),thetwoequationscouldberewrittenas(1) and(2)12 USINGTHECLASSIFIERS upposewe vetrainedaclassifierfrominputdata,,andth eircorrespondingexpert-labelledoutputdat a.

9 AndsupposethatintheprocessweformedsomeGP outputscorrespondingtothesedata, renowreadytoinputanewdatapoint,,inthelef tsideofourschematic, ,findingtheprobabilityissimilartoGPR, (2):(3)(willbeexplainedsoon,butfornowcon siderittobeverysimilarto.)Inthesecondste p,wesquashtofindtheprobabilityofclassmem bership,.Theexpectedvalueis(4)Thisisthei ntegralofacumulativeGaussiantimesaGaussi an, (2006),thesolutionis:(5) ,sothatweknoweverythingabouttheGPpro-duc ing(3), ,naturallywe ,howlikelytheyaretobeappropriateforthetr ainingdatacanbedecomposedusingBayes theorem:(6)Let ,(7)Droppingthesubscriptsintheproduct,is informedbyoursigmoidfunction.

10 Specifically,isbydefinition,andtocomplet ethepicture,. ,butfirstwe 20 1001020 10 8 6 4 20246810xLatent function f(x) 20 probabilityFigure1:(a)Toyclassificationd ataset,wherecirclesandcrossesindicatecla ssmem-bershipoftheinput(training)data, , (mean)probabilityprob, answer toourproblemaftersuccessfullyperformingG PC.(b)Thecorrespondingdistributionofthel atentfunction, (6)withrespecttoiszero,orequivalentlyand moresimply, ,andusingthesamelogicthatproduced(10)int hepreviousdocument,wefindthat(8) ,appearsonbothsidesoftheequation,sowemak eaninitialguess(zeroisfine) (8)canbeuseddirectlyin(3),sowe (6),whichturnsouttobe, ,wepretendisGaussiandistributed, (9)(Thisassumptionisoccasionallyinaccura te,soifityieldspoorclassifications,bet-t erwaysofcharacterizingtheuncertaintyinsh ouldbeconsidered,forexampleviaexpectatio npropagation.)