Practical Byzantine Fault Tolerance

1 AppearsintheProceedingsoftheThird SymposiumonOperatingSystemsDesignandImpl ementation,New Orleans,USA,February1999 PracticalByzantineFaultToleranceMiguelCa stroandBarbaraLiskovLaboratoryforCompute rScience,MassachusettsInstituteofTechnol ogy,545 TechnologySquare, Cambridge, ,thealgorithmdescribedinthispaperis Practical :it worksinasynchronousenvironmentslike theInternetandincorporatesseveralimporta ntoptimizationsthatimprove thatourserviceis only3%slowerthana ,thenumberofsoftwareerrorsis ( ,arbitrary)behavior, new,practicalalgorithmforstatemachinerep lication[17, 34] totalofreplicasaresimultaneouslyfaulty. Thismeansthatclientseventuallyreceive repliestotheirrequestsandthoserepliesare correctaccordingto linearizability[14, 4].

2 Thealgorithmworksinasynchronoussystemsli ke significantbodyofworkonagreementThisrese archwas supportedin partbyDARPA under contract DABT63-95-C-005,monitoredbyArmyFortHuach uca,andundercontractF30602-98-1-0237,mon itoredbytheAirForceResearchLaboratory,an din partially supportedbya (startingwith[19]).However, mostearlierwork( ,[3, 24, 10])eitherconcernstechniquesdesignedtode monstratetheoreticalfeasibilitythatareto oinefficienttobeusedinpractice,orassumes synchrony, , ,Rampart[30] andSecureRing[16], weredesignedtobepractical,buttheyrelyont hesynchrony assumptionforcorrectness, servicebydelayingnon-faultynodesorthecom municationbetweenthemuntilthey denial-of-serviceattackis generallyeasierthangainingcontrolovera , ,itusesanefficientauthenticationschemeba sedonmessageauthenticationcodesduringnor maloperation.

3 Public-keycryptography, whichwascitedasthemajorlatency [29]andthroughput[22] bottleneckinRampart,is evaluateourapproach,weimplementeda replica-tionlibraryandusedit toimplementa usedtheAndrewbench-mark[15] to thatoursystemis only3% ,thepapermakesthefollowingcontributions: It describesa numberofimportantoptimizationsthatallow thealgorithmtoperformwellsothatit describestheimplementationofa beginbydescribingoursystemmodel, concludewithasummaryofwhatwehave accomplishedanda assumeanasynchronousdistributedsystemwhe renodesareconnectedbya ,delaythem,duplicatethem, usea Byzantinefailuremodel, ,faultynodesmaybehave arbitrarily, subjectonlytotherestrictionmentionedbelo w.

4 We thisassumptionto betruein thepresenceof maliciousattacks,somestepsneedtobetaken, ,eachnodeshouldrundifferentimplementatio nsoftheservicecodeandoperatingsystemands houldhave a differentrootpasswordanda [28] andforlow , ,differentteamsofprogrammersproducediffe rentimplementations,is signatures[33],messageauthenticationcode s[36], andmessagedigestsproducedbycollision-res istanthashfunctions[32].We denoteamessagesignedbynodeasandthedigest ofmessageby. We followthecommonpracticeofsigninga digestofa messageandappendingit totheplaintextofthemessageratherthansign ingthefullmessage(shouldbeinterpretedint hisway).Allreplicasknow theothers publickeysto ,delaycommunication, doassumethattheadversarycannotdelaycorre ctnodesindefinitely.

5 We alsoassumethattheadversary(andthefaultyn odesitcontrols)arecomputationallyboundso that(withveryhighprobability) ,theadversarycannotproducea validsignatureofa non-faultynode,computetheinformationsumm arizedbya digestfromthedigest,orfindtwo havetheseproperties[33, 36, 32].3 ServicePropertiesOuralgorithmcanbeusedto implement any deterministicreplicatedservicewithastate andsomeoperations. Theoperationsarenotrestrictedtosimplerea dsorwritesofportionsoftheservicestate;th ey thereplicatedservicetoinvoke operationsandblockwaitingfora reply. Thereplicatedserviceis they followthealgorithminSection4 andif Safetymeansthatthereplicatedservicesatis fieslinearizability[14](modifiedtoaccoun tforByzantine-faultyclients[4]):itbehave slike a centralizedimplementationthatexecutesope rationsatomicallyoneata faultyreplicacanbehave arbitrarily, ,it candestroy (evenif they colludewithfaultyreplicas):alloperations performedbyfaultyclientsareobservedina , iftheserviceoperationsaredesignedtoprese rve someinvariantsontheservicestate, , ,ina filesystema , welimittheamountofdamagea faultyclientcandobyprovidingaccesscontro l.

6 Weauthenticateclientsanddeny accessif theclientissuinga requestdoesnothavetherighttoinvoke ,servicesmayprovideoperationstochangethe accesspermissionsfora ,thisprovidesa powerfulmechanismto toprovidesafety. Therefore,it mustrelyonsynchrony toprovideliveness;otherwiseitcouldbeused toimplementconsensusinanasynchronoussyst em,whichisnotpossible[9].Weguaranteelive ness, ,clientseventuallyreceive repliestotheirrequests, ,delayisthetimebetweenthemomentwhena messageis sentforthefirsttimeandthemomentwhenit is receivedbyitsdestination(assumingthesend erkeepsretransmittingthemessageuntilit is received).(Amoreprecisedefinitioncanbefo undin[4].)Thisisa ratherweaksynchronyassumptionthatislikel ytobetrueinany realsystemprovidednetworkfaultsareeventu allyrepaired,yetitenablesustocircumventt heimpossibilityresultin[9].

7 Theresiliency of ouralgorithmis optimal:31 is theminimumnumberofreplicasthatallow anasynchronoussystemto providethesafetyandlivenesspropertieswhe nuptoreplicasarefaulty(see[2] fora proof).Thismany replicasareneededbecauseit mustbepossibletoproceedaftercommunicatin gwithreplicas, ,it is possiblethatthereplicasthatdidnotrespond arenotfaultyand,therefore,ofthosethatres pondedmightbefaulty. Evenso,theremuststillbeenoughresponsesth atthosefromnon-faultyreplicasoutnumberth osefromfaultyones, ,2. :a faultyreplicamayleakinformationtoanattac ker. It is notfeasibleto offerfault-tolerantprivacyinthegeneralca sebecauseserviceoperationsmayperformarbi trarycomputations usingtheirargumentsandtheservicestate;re plicasneedthisinformationinthecleartoexe cutesuchoperationsefficiently.

8 It is possibletousesecretsharingschemes[35] toobtainprivacy eveninthepresenceofa thresholdofmaliciousreplicas[13] a formofstatemachinereplication[17,34]:the serviceismodeledasa statemachinethatisreplicatedacrossdiffer entnodesina denotethesetofreplicasbyandidentifyeachr eplicausinganintegerin01. Forsimplicity, weassume31 whereisthemaximumnumberofreplicasthatmay befaulty;althoughtherecouldbemorethan31 replicas,theadditionalreplicasdegradeper formance(sincemoreandbiggermessagesarebe ingexchanged) througha successionofconfigura-tionscalledviews. Ina viewonereplicais theprimaryandtheothersarebackups. Viewsarenumberedcon-secutively. Theprimaryofa viewis replicasuchthatmod, [26] andPaxos[18]useda similarapproachtotoleratebenignfaults(as dis-cussedinSection8.)

9 Clientsendsa requesttoinvoke a repliesfromdifferentreplicaswiththesamer esult; allstatemachinereplicationtechniques[34] ,weimposetwo requirementsonreplicas:they mustbedeterministic( ,theexecutionofanoperationina givenstateandwitha givensetofargumentsmustalwaysproducethes ameresult)andthey muststartin requirements,thealgorithmensuresthesafet ypropertybyguaranteeingthatallnon-faulty replicasagreeona ,weassumethatmessageauthenticationisachi evedusingdigitalsignaturesratherthanthem oreefficientschemebasedonmessageauthenti cationcodes; [21] is presentedin[4]. s requestsaretotallyorderedsuchthatlaterre questshave highertimestampsthanearlierones;forexamp le,thetimestampcouldbethevalueoftheclien t s localclockwhentherequestis theclientincludesthecurrentview number, requesttowhatitbelievesisthecurrentprima ryusinga thecurrentview number,is thetimestampofthecorrespondingrequest,is thereplicanumber, andis replieswithvalidsignaturesfromdifferentr eplicas,andwiththesameand, before3acceptingtheresult.

10 Thisensuresthattheresultis valid,sinceat repliessoonenough, therequesthasalreadybeenprocessed,therep licassimplyre-sendthereply;replicasremem berthelastreplymessagethey sentto ,if thereplicais nottheprimary,it relaystherequesttotheprimary. If theprimarydoesnotmulticasttherequesttoth egroup,it willeventuallybesuspectedtobefaultybyeno ughreplicastocauseaview clienttomake asynchronousrequests,yetpreserve ,amessage logcontainingmessagesthereplicahasaccept ed,andanintegerdenotingthereplica s currentview. We describehow to truncatethelogin ,, receivesa clientrequest,,it startsa ,it grouptocutdownonmessagetrafficandCPUover headsunderheavyload; thisoptimizationis similarto a groupcommitin transactionalsystems[11].