Transcription of Introduction to Machine Learning - dl.matlabyar.com
1 I n t r o d u c t i o nt oM a c h i n eL e a r n i n gEthemAlpayd nTheMITP ressS o l u t i o n s M a n u a PrintedonJanuary10, :Youcanfaxa document,thatis,sendtheimage,oryoucanuse anopticalcharacterreader(OCR)andsendthet ext letypicallyis shorterthantheimage lebuta faxeddocu-mentcanalsocontaindiagrams,pic tures, ,welosepropertiessuchasfont,size,etc(unl esswealsorecognizeandtransmitsuchinforma tion)orthepersonaltouchif it is ,andforambigiouscases, faxma-chineis cheaperandeasierto ndthana goodif wehavehighvolume,goodqualitydocuments;fo rdoc-umentsoffewpageswithsmallamountofte xt,it is ,westorethebitmapof thatcharacterasa systemallowsonlyonetemplatepercharactera ndcannotdis-tinguishcharactersfrommultip lefonts, ,thefontsyoutypicallyseein vouchersandbankingslips,whichareusedwith OCRsoftware, better(cheaper,morereliable,moreavailabl e) ina junke-mailthatletsusknowthatit is junk?
2 Howcanthecomputerdetectjunkthrougha syntacticanalysis?Whatwouldyoulikethecom putertodoif it detectsa junke-mail deleteit automatically,moveit toa di erent le,orjusthighlightit onthescreen?Typically,spam opportunity , viagra , dollars aswellascharacterssuchas'$','!' trainingsetofexamplepastemailsthattheuse rhaspreviouslymarkedasspam(Oneveryfreque ntlyusedmethodforspam lteringis thenaiveBayes'classi ).Thespam ltersdonotworkwith100percentreliabilitya ndfre-quentlymakeerrorsinclassi a junkmailisnot lteredandshowedtotheuser,thisis notgood,butit is notasbadas lter-inga ,mailmessagesthatthesystemconsidersasspa mshouldnotbeautomaticallydeletedbutkepta sidesothattheusercanseethemif he/shewantsto,especiallyintheearlystages ofusingthespam lterwhenthesystemhasnotyetbeentrainedsu lteringspamwillprobablyneverbesolvedcomp letelyasthespammerskeep ndingnovelwaystooutdothe lters.
3 Theyusedigit`0'insteadoftheletter'O',dig it`1'insteadofletter`l'topassthewordtest s,addpiecesoftextsfromregularmessagesfor themailtobeconsiderednotspam,orsendit asimagenotastext(andlatelydistorttheimag ein smallrandomamountstothatit is notalwaysthesameimage).Still,spam lteringis theoutput?Howcanyoucommunicatewiththepas senger?Doyouneedtocommunicatewiththeothe rautomatedtaxis,thatis,doyouneeda language ?3 Anautomatedtaxishouldbeabletopicka passengeranddrivehim/hertoa shouldhavesomepositioningsystem(GPS/GIS) andshouldhaveothersensors(cameras)tobeab letosensecars,pedes-trians, corscheduling,loadbalancing, ,wewantto ndthedependencebetweentwoitemsXandY.
4 Givena databaseofcustomertransactions,howcanyou ndthesedependencies?Howwouldyougeneraliz ethistomorethantwoitems?Thisis beannoying? , ndsSandGfroma givenin C S `+'/'o' ,SandGaretheactualconcept,themostspeci time, case?(Hint:Seethecandidateeliminationalg orithminMitchell1997.) ,SandGareupdatedasfollows(Mitchell,1997; ): Ifxisa positiveexample,removeanyg2 Gthatcoversxandexpandanys2 Sthatdoesnotcoverx Ifxisa negativeexample,removeanys2 Sthatcoversxandrestrictanyg2 GthatdoescoverxTheimportantpointis thatwhenwearerestrictingag2G(special-iza tion)orexpandingas2S(generalization),the remaybemorethanonewayofdoingit,andthiscr eatesmultiplehypothesesinSorG.
5 Forexample,in ,if weseea negativeexampleat 20000;2000 aftertwopositiveexamplesG 1< x <1; 1< y <1 splitsintwo:G 1< x <20000; 1< y <1 ; 1< x <1; 1< y <2000 . Thesearetwodi erentwaysofspecializingGsothatit 2 : Engine power x 1 : Price G 1 10,000 20,000 15,000 1,400 1,200 1,600 1,800 S G 2 it bettertousetheaverageofSandGasthe nalhypothesis?If thereisnoise,instancesmaybeslightlychanged;insucha case, a circleinsteadofa circlehypothesisbecalculatedinsucha case?Whatif it is anellipse?Whydoesit makemoresenseto useanellipseinsteadof a circle?HowcanyougeneralizeyourcodetoK >2classes?In thecaseofa circle,theparametersarethecenterandthera dius(see ).
6 Wethenneedto ndthetightestcirclethatincludesallthepos itiveexamplesasSandGwillbethelargestcirc lethatincludesallthepositiveexamplesandn onegativeexample:c x 2 : Engine power x 1 : Price C r c c 1 c 2 a circlewithtwoparameters, makesmoresensetouseanellipsebecausethetw oaxesneednothavethesamescaleandanellipse hastwoseparateparametersforthewidthsinth etwoaxesratherthana >2 classes,weneeda , therewillbeonehypothesiswhichtakes7allel ementsofCiaspositiveexamplesandinstances ofallCj;j6 notonerectanglebuta unionoftwo(orm >1) theadvantageofsucha hypothesisclass?Showthatanyclasscanberep resentedbysucha a singlerectangle,allthepositiveinstancess houldformonesinglegroup;byincreasingthen umberofrectangles,weget (see ),thepositiveinstancescanformtwo, conjunctiononthetwoinputattributesandhav ingmultiplerectangles,correspondstoa (m N), wecanhavea 2 : Engine power x 1 : Price h 1 h 2 C a wehavea supervisorwhocanprovideuswiththelabelfor anyx,whereshouldwechoosextolearnwithfewe rqueries?
7 Theregionofambiguityis betweenSandG. It a giveninstancethereturnsouttobepositive,t hismeanswecan82 SupervisedLearningmakeSlargeruptothatins tance;if it is negative, ,wesummedupthesquaresof thedi theonemostfrequentlyused,butit is sumsupthesquaresofthedi erences,it is bettererrorfunctiontoimplementrobustregr ession?AsweseeinChapter4, thenoisecomesfroma distributionwithlongtails,thensummingups quareddi erencescausea few,farawaypoints, ,outliers,tocorruptthe ectofoutliers,wecansumuptheabsolutevalue ofdi erencesinsteadofsquaringthem:E gjX 1 NNXt 1jrt g xt jbutnotethatwelosefromdi ( )whichusesabsolutedi erenceandalsohasa termthatneglectstheerrorduetoverysmalldi ,setthemequalto0 andsolvethesetwoequationsintwounknowns:E w1.
8 W0jX 1 NNXi 1 rt w1xt w0 2@E@w0 Xt rt w1xt w0 0 Xtrt w1 Xtxt Nw0w0 Xtrt=N w1 Xtxt=N r w1x@E@w1 Xt rt w1xt w0 xt 09 Xtrtxt w1Xt xt 2 w0 XtxtXtrtxt w1Xt xt 2 r w1x XtxtXtrtxt w10@Xt xt 2 xXtxt1A rXtxtXtrtxt w10@Xt xt 2 xNx1A rNxw1 Ptrtxt xrNPt xt 2 thesetoflines,andweusea linetoseparatethepositiveandnegativeexam ples,insteadofboundingthepositiveexample sasina rectangle,leavingthenegativesoutside(see ).ShowthattheVCdimensionofa lineis ,forallpossiblelabelingofthreepoints,the reexista ,nomatterhowweplacethesefourpointsintwod imensions,thereis atleastonelabelingwherewecannotdrawa 1 x 2 All possible labelings of three points can be separated using a line.
9 X 1 x 2 These four points cannot be separated using a line. line, 7 intwo102 SupervisedLearningdimensions.(Hint:Forbe stseparation,it is bestto placethesevenpointsequidistantona circle.)Aswecanseein ,forallpossiblelabelingofsevenpoints,wec andrawa 1 x 2 x 1 x 2 These eight points with this labeling cannot be separated using a triangle. These seven points can be separated using a triangle no matter how they are labeled. two-classproblem,thelikelihoodratioisp xjC1 p xjC2 nea discriminantfunctionasg x P C1jx P C2jx andchoose(C1ifg x >1C2otherwiseWecanwritethediscriminantas theproductofthelikelihoodratioandtherati oofpriors:g x p xjC1 p xjC2 P C1 P C2 If thepriorsareequal,thediscriminantis thelikelihoodratio(seealsothenextexercis e).)
10 Two-classproblem,thelogoddsis de nedaslogP C1jx P C2jx nea discriminantfunctionasg x logP C1jx P C2jx andchoose(C1ifg x >0C2otherwise123 BayesianDecisionTheoryLogoddsis thesumofloglikelihoodratioandlogofpriorr atio:g x logp xjC1 p xjC2 logP C1 P C2 If thepriorsareequal,thediscriminantis two-class,two-actionproblem,if thelossfunctionis 11 22 0, 12 10, and 21 1, :R 1jx 11P C1jx 12P C2jx 10P C2jx R 2jx 21P C1jx 22P C2jx P C1jx andwechooseC1ifR 1jx < R 2jx , orifP C1jx >10P C2jx ,P C1jx >10= faircoinandif theresultis heads,yougetnothing,otherwiseyouget$ thewinis $500insteadof$5?With vedollars,theexpectedearningis 1=2 0 1=2 5 2 , ,inthesensethatthoughtheymayrisksmallamo unts,theydonotliketorisklargeramounts, , , Theframingofdecisionsandthepsychologyofc hoice, Science211:453 ,calculateP CjW.)
