Honey, I Shrunk the Sample Covariance Matrix - Ledoit

1 Honey, I ShrunktheSampleCovarianceMatrixOlivierLe doitEquitiesDivisionCreditSuisseFirstBos tonOneCabot Department of EconomicsandBusinessUniversitatPompeu 2003 AbstractThecentralmessageof thispaper is thatnobodyshouldbe usingthesamplecovariancematrixforthepurp oseof containsestima-tionerrorof thekindmostlikelyto perturba ,we suggestusingthematrixobtainedfromthesamp lecovariancematrixthrougha transformationcalledshrinkage. Thistendsto pullthemostextremecoe cients towardsmorecentralvalues,thereby systematicallyreducingestima-tionerrorwh ereit , thechallengeis to know theoptimalshrinkageintensity, andwe give otherstepin theportfoliooptimizationprocess,we show onactualstock market datathatshrinkagereducestrackingerrorrel ative to a benchmarkindex,andsubstan-tiallyincrease stherealizedinformationratioof theactive portfoliomanager. Phone:+34-93-5422552,Fax:+34-93-54217461 1 IntroductionSincetheseminalworkof Markowitz(1952),mean-varianceoptimizatio nhasbeenthemostrigorousway to pick stocks in which to fundamentalingredientsaretheexpected(exc ess)returnforeach stock, which represents theportfoliomanager'sability to forecastfuturepricemovements,andthecovar iancematrixof stock returns,which represents furtherspecifytheproblem,in therealworldmostassetmanagersareforbidde nfromsellingany stock short,andin themodernworldtheyaretypicallymeasuredag ainstthebenchmarkof anequity market indexwith xed(orinfrequentlyrebalanced) thisproblem|providedtheyarefedtheright inputs, stock returnshasalways beenoneof is to gathera historyof paststock returnsandcomputetheirsamplecovariance Matrix .

2 Unfortunatelythiscreatesproblemsthatarew elldocumented(JobsonandKorkie,1980).To putit assimplyaspossible,whenthenumber of stocks underconsiderationis large,especiallyrelative to thenumber ofhistoricalreturnobservationsavailable( which is theusualcase),thesamplecovariancematrixi s estimatedwitha lotof impliesthatthemostextremecoe cients in thematrixthusestimatedtendto take onextremevaluesnotbecausethisis \thetruth",butbecausetheycontainanextrem eamount of onto themandplaceits biggestbetson thosecoe cientswhich (1989)callsthisphenomenon\error-maximiza tion".It impliesthatmanagers'realizedtrack recordswillunderrepresent theirtruestock-pickingabilities,which is of this,somecompaniessuch as APTandBARRA have proposedpro-prietarymethods to generatecovariancematricesthatareadverti zedas arethatany managerusingthemestablishesa costlyandinde nitedependenceonanexternalentity whodoes notsharein any downsiderisk,andthattheirproprietarymeth ods arenotopenforindependent inspectionandveri cation,so onecannever be why we proposea newformulaforestimatingthecovariancematr ixof stockreturnsthatcanbene ciallyreplacethesamplecovariancematrixin any mean-varianceoptimizationapplication,and is absolutelyfreeof chargeandopento everybody.

3 Thecruxof themethod is thatthoseestimatedcoe cients in thesamplecovariancematrixthatareextremel yhightendto containa lotof positive errorandthereforeneedto bepulleddownwardsto , we compensateforthenegativeerrorthattendsto be embeddedinsideextremelylow estimatedcoe cients by callthistheshrinkageof properlyimplemented,thisshrinkagewouldcl early xtheproblemof thesamplecovariancematrixdescribedabove. Thekey questionsaretowardswhattargetto shrink,andhowintensely?Ourcontributionsa re:(1)to providea rigorousstatisticalanswer to boththesequestions;(2)to describe thisbelow so thereadercandecideforhimor herselfwhetherit makes sense;(3)tosupplycomputercodethatimpleme nts theresultingmathematicalformula;and nally(4)to show thatit yieldssigni cant improvements onactualstock hardlya newandrevolutionaryconceptin Statistics,althoughit cer-tainlywas when rstintroducedby ProfessorCharlesSteinof StanfordUniversity non-technicalprimeronshrinkageusingreal- lifeexamplesof base-ballbattingaverageswas writtenby EfronandMorris(1977).

4 Thatthisideahasnotyet percolatedtoa eldwhereit wouldbe mostuseful,portfoliomanagement, is atestimony thatChinesewallsstillexistbetweentheoret icalandapplieddisciplineswhowouldbene tfromtalkingto each endeavor to knock useshrinkagein portfolioselectionweremadeby FrostandSavarino(1986)andJorion(1986),bu ttheirparticularshrinkagetechniquesbroke downwhenthenumber of stocks onthemenu exceedsthenumber of historicalreturnobservations,which is veryoftenthecasein (2003)provedthatmean-varianceoptimizersa realreadyimplicitlyapplyingsomeformof shrinkagetothesamplecovariancematrixwhen shortsalesareruledout,andthatthisis generallybene cialin termsof improvingweights stability. Allthemorereasonthento doitexplicitlyso thattheoptimalshrinkageintensity canbe applied,as in thefoundationsforthepresent workhasbeenlaidoutby theauthorsin otherpapers(LedoitandWolf,2003,2004).Tho sewerelargelytheoreticalandgeneral-inter estar-ticles,whereasthepresent onefocusesspeci callyonhow to employ thetechnologytoaddvalueto active give a formaldescriptionof theportfoliooptimizationproblemin ordertoprovidea ,we thenlookat itsout-of-sampleperformance,usinghistori calstock FormalDescriptionof the ProblemWe studythemostrelevant caseforequity a weightedindexof a largenumberNof individualstocks,such as a stocksfromwhich nedrelative to nethefollowingnotations:wB=vectorof benchmarkweights fortheuniverseofNstocksx=vectorof active weightswP=wB+x=vectorof portfolioweightsy=vectorof stock returns = E(y) =vectorof expectedstock returns = w0B =vectorof expectedstock excessreturns =covariancematrixof stock returnsWe canwriteexpectedreturnsandvariancesin vector/matrixnotationas.

5 B=w0B =expectedreturnonbenchmark 2B=w0B wB=varianceof benchmarkreturn P=w0P =expectedreturnonportfolio 2P=w0P wP=varianceof portfolioreturn E=x0 =expectedexcessreturnonportfolio 2E=x0 x=trackingerrorvarianceTheportfolioselec tionproblemis subjectto theconstraint thattheportfoliobe fullyinvested,thatis,theportfolioweights wPhave to addupto unity. With1denotinga conformingvectorof ones,thiscanbe writtenasw0P1= alsoadduptounity, thevectorof portfoliodeviationsmustuptozero,orx01= ,theportfolioof themanagercanbe viewedas a positionin thebenchmarkplusanactiveportfolio. Theactive portfoliois a long/shortportfolioandexpressestheviewso f immediateimplicationsare: P= B+ E 2P= 2B+ 2w0B x+ 2 EWhilepositionsof theactive portfolioarebothpositive andnegative, themanagerdoes nothave theportfolioweightswPcanbe negative, orwP 0, dueto thelong-onlyconstraint. Theresultingconstraintx wBexpressesthelimitedfreedomof (2000,Chapter15)illustratehowthislimitat ioncannegativelya ecttheperformanceof themanagedportfolio,especiallywhentheben chmarkis a value-weightedindexandwhenNis addition,themanageroftenis facedwiththeconstraint thatthetotalpositionin any givenstock4cannotexceeda certainvalue,like 10%.

6 If thisupper boundis denotedbyc, theresultingconstraint ontheweights of theactive portfolioisx c1 nedthevariousingredients,we cannow formalizetheoptimizationprob-lemof themanageras follows:2 Minimize:x0 x(1)such that:x0 gx01= 0x wBx c1 wBHeregis themanager'stargetgain( ,expectedexcessreturn)relative to typicalnumber is 300basispoints (annualized).Themanagerchoosesgandtheupp er limitcandalsoknowsthecurrent vectorof benchmarkweightswB. Sheisnow leftto provideestimatesfor , thevectorof expectedstock excessreturns,andfor ,thecovariancematrixof stock a nalstep,alltheinputsarefedintoa quadraticoptimizationsoftwarethatwillcom putex, theoptimalweights of is ourmissionto providethemanagerwitha good estimatorof .We donothelpwiththeproblemof how to estimate .3 Shrinkage Estimatorof the PrincipleThissectionbrie ydescribes theshrinkageestimatorfor which we propose;fora moredetailedexpositionandrelevant historicalbackgroundthereaderis referredto LedoitandWolf(2003).

7 We startwiththesamplecovariancematrixS. Itsadvantagesareeaseof computationandtheproperty of beingunbiased( ,itsexpectedvalueis equalto thetruecovariancematrix).Itsmaindisadvan tageis thefactthatit containsa lotof estimationerrorwhenthenumber of datapoints is of comparableor evensmallerorderthanthenumber of individualstocks;thisis thecommonsituationin , onemight consideranestimatorwitha lotof structure,as thesingle-factormodelof Sharpe (1963).Such estimatorscontainrelativelylittleestimat ionerrorbut,ontheotherhand,tendto be misspeci edandcanbe oneway or another,allsuccessfulriskmodels nda compromisebetweenthesamplecovariancematr ixanda to incorporatemultiple5factorsinsteadof justthesinglefactorof Sharpe (1963).Thereby themodelsbecomemore exibleandtheirbiasis by decidingonthenatureandthenumber of thefactorsincludedinthemodelis as much anartas it is a is to usea combinationofindustryfactorsandriskindic es, withthetotalnumber of factorsbeingontheorderof BARRA' is to usestatisticalfactors, such as principalcomponents,withthetotalnumber of factorsbeingon theorderof 5.

8 A commercialvendoro eringriskmodelsbasedonstatisticalfactors is is di erent. ConsiderthesamplecovariancematrixSanda highlystructuredestimator,denotedbyF. We nda compromisebetweenthetwo by com-putinga convex linearcombination F+ (1 )S, where is a number between0 calledshrinkage, sincethesamplecovariancematrixis ` Shrunk ' is referredto as , it measurestheweight thatis givento a longandsuccessfulhistoryin of theprincipleis thatby properlycombiningtwo `extreme'estimatorsonecanobtaina `compromise' make a somewhatsloppy anal-ogy:mostpeoplewouldbe preferthe`compromise'of onebottleof Bordeauxandonesteakto either`extreme'of two bottlesof Bordeaux(andnosteak)or two steaks(andnoBordeaux).Any shrinkageestimatorhasthreeingredients:An estimatorwithnostructure,anestimatorwith a lotof structure,anda shrinkageconstant. Theestimatorwithoutstructureis generallyquiteobvious,given us it is thestructuredestimator,or shrinkagetarget, TargetTheshrinkagetargetshouldful lltwo requirements at thesametime:it involves onlyasmallnumber of freeparameters(thatis, a lotof structure)butit alsore ectsimportantcharacteristicsof theunknownquantity (2003)suggestthesingle-factormatrixof Sharpe (1963)as thispaper we makea di erent suggestion:theconstantcorrelationmodel.

9 In ourexperience,it gives com-parableperformancebutis easierto implement. Themodelsays thatallthe(pairwise) themodelis allthesamplecorrelationsis theestimatorof thecommonconstant togetherwiththevectorof samplevariancesimpliesourshrinkagetarget ,de-notedbyFin theremainderof formaldescriptionof theshrinkagetarget6is providedin AppendixA;in particular,seeequation(3). ConstantTheobviouspracticalproblemis which valueto choosefortheshrinkageconstant. Anychoiceof strictlybetween0 and1 wouldyielda compromisebetweenSandF. Butthisresultsin in nitelymany , thereis an`optimal' is . AppendixB derivesa formulaforestimating . Theestimatedoptimalshrinkageconstant is denoted^ ;seeequation(5)in thecovariancematrix is now readyforuse5:^ Shrink=^ F+ (1 ^ )S(2)4 EmpiricalStudyWe now studytheout-of-sampleperformanceof ourshrinkageestimator,usinghistoricalsto ck market usethesedatatoconstructseveralvalue-weig htedindicestoserve ,themethodologyis thebeginningof each month,weselecttheNlargeststocks as measuredby theirmarket valuesofthestocksde theendof themonth,we observe the(real)returnsof theindividualstocksand,giventheirweights , repeatedeverymonth until theendof December ,theconstituents listandtheindexweights thebenchmarksizeNis concerned,we employN= 30;50;100;225.

10 Important benchmarksas DJIA,XetraDAX,DJSTOXX50,FTSE100,NASDAQ-1 00,NIKKEI225,andS& thevariousbenchmarkreturnsareprovidedin mimica skilledactive manager,we rstconstructrawforecastsof theexpectedexcessreturnsby secondstep,theserawforecastsaretransform edintore nedforecasts^ which arefedto donein a way such thattheunconstrainedannualizedexanteinfo rmationratio(IR)is approximatelyequalto ,independentlyof thevalueofthebenchmarksizeN. TheunconstrainedIRcouldbe attainedby a managerwhodidnotfaceany lower or upper boundconstraints ontheweight vectorxandwhoknew7theexactnatureof thecovariancematrix of stock theforecastconstructionaredescribedin riskmodelis evaluatedby : At thebeginningof each monthfeedthefollowingingredients to thequadraticoptimizer:thebenchmarkweight swB, theforecastedexpectedexcessreturns^ ,theestimatedcovariancematrix^ ,thedesiredgaing, andanupper boundofc= 0:1 onthetotalweight of any stock.