COVER FEATURE An Introduction to Evaluating Biometric …

1 0018-9162/00/$ 2000 IEEE56 ComputerAn Introduction toEvaluating Biometric SystemsOn the basis of media hype alone, you mightconclude that Biometric passwords will soonreplace their alphanumeric counterpartswith versions that cannot be stolen, forgot-ten, lost, or given to another person. Butwhat if the performance estimates of these systems arefar more impressive than their actual performance?To measure the real-life performance of biometricsystems and to understand their strengths and weak-nesses better we must understand the elements thatcomprise an ideal Biometric system. In an ideal system all members of the population possess the char-acteristic that the Biometric identifies, like irisesor fingerprints; each Biometric signature differs from all others inthe controlled population; the Biometric signatures don t vary under the con-ditions in which they are collected; and the system resists evaluation quantifies how well bio-metric systems accommodate these , Biometric evaluations require that an inde-pendent party design the evaluation, collect the testdata, administer the test, and analyze the designed this article to provide you with suffi-cient information to know what questions to ask whenevaluating a Biometric system, and to assist you indetermining if performance levels meet the require-ments of your application.

2 For example, if you plan touse a Biometric to reduce as opposed to eliminate fraud, then a low-performance Biometric system maybe sufficient. On the other hand, completely replacingan existing security system with a Biometric -based onemay require a high-performance Biometric system, orthe required performance may be beyond what cur-rent technology can we focus on Biometric applications that givethe user some control over data acquisition. Theseapplications recognize subjects from mug shots, pass-port photos, and scanned fingerprints. Examples notcovered include recognition from surveillance photosor from latent fingerprints left at a crime the biometrics that meet these constraints, voice,face, and fingerprint systems have undergone the moststudy and testing and therefore occupy the bulk ofour discussion. While iris recognition has receivedmuch attention in the media lately, few independentevaluations of its effectiveness have been STATISTICST here are two kinds of Biometric systems: identifi-cationand identification systems, a Biometric signature ofan unknown person is presented to a system.

3 The sys-tem compares the new Biometric signature with a data-base of Biometric signatures of known individuals. Onthe basis of the comparison, the system reports (or esti-mates) the identity of the unknown person from thisdatabase. Systems that rely on identification includethose that the police use to identify people from fin-gerprints and mug shots. Civilian applications includethose that check for multiple applications by the sameperson for welfare benefits and driver s verification systems, a user presents a biometricsignature and a claim that a particular identity belongsto the Biometric signature. The algorithm either acceptsHow and where Biometric systems are deployed will depend on their performance. Knowing what to ask and how to decipher the answers canhelp you evaluate the performance of these emerging JonathonPhillipsAlvin WilsonMarkPrzybockiNationalInstitute ofStandards andTechnologyCOVER FEATUREor rejects the claim.

4 Alternatively, the algorithm canreturn a confidence measurement of the claim s valid-ity. Verification applications include those that authen-ticate identity during point-of-sale transactions or thatcontrol access to computers or secure statistics for verification applicationsdiffer substantially from those for identification sys-tems. The main performance measure for identificationsystems is the system s ability to identify a biometricsignature s owner. More specifically, the performancemeasure equals the percentage of queries in which thecorrect answer can be found in the top few example, law enforcement officers often use anelectronic mug book to identify a suspect. The inputto an electronic mug book is a mug shot of a suspect,and the output is a list of the top matches. Officers maybe willing to examine only the top twenty such an application, the important performancemeasure is the percentage of queries in which the cor-rect answer resides in the top twenty performance of a verification system, on theother hand, is traditionally characterized by two errorstatistics: false-reject rate and false-alarm rate.

5 Theseerror rates come in pairs; for each false-reject rate thereis a corresponding false alarm. A false reject occurswhen a system rejects a valid identity; a false alarmoccurs when a system incorrectly accepts an a perfect Biometric system, both error rateswould be zero. Unfortunately, Biometric systemsaren t perfect, so you must determine what trade-offsFebruary 200057 Biometric OrganizationsKirk L. Kroeker, ComputerAlthough poised for substantial growth as the marketplace beginsto accept biometrics, recent events have demonstrated that thefledgling industry s growth could be severely constricted by mis-information and a lack of public particular, concerns about privacy can lead to ill-informedregulations that unreasonably restrict biometrics use. The lack ofcommon and clearly articulated industry positions on issues suchas safety, privacy, and standards further increase odds that gov-ernments will react inappropriately to uninformed and evenunfounded assertions regarding Biometric technology s functionand organizations, the International Biometric IndustryAssociation and the Biometric Consortium, aim to improve thissituation.

6 International Biometric Industry AssociationA Washington, trade association, the IBIA seeks togive the young industry a seat at the table in the growing publicdebate on the use of Biometric technology. The IBIA focuses oneducating lawmakers and regulators about how biometrics canhelp deter identity theft and increase personal addition to helping provide a lobbying voice for biometriccompanies, the IBIA s board of directors has taken steps to estab-lish a strong code of ethics for its members. In addition to cer-tifying that the consortium will adhere to standards for productperformance, each member must recognize the protection ofpersonal privacy as a fundamental obligation of the promoting a position on member ethics, the IBIA rec-ommends safeguards to ensure that Biometric data is not misused tocompromise any information; policies that clearly set forth how Biometric data will be col-lected, stored, accessed, and used; limited conditions under which agencies of national securityand law enforcement may acquire, access, store, and use bio-metric data; and controls to protect the confidentiality and integrity of data-bases containing Biometric IBIA is open to Biometric manufacturers, integrators, andend users ( ).

7 Biometric ConsortiumOn 7 December 1995, the Facilities Protection Committee (acommittee of the Security Policy Board established by US PresidentBill Clinton) chartered the Biometric Consortium. With more than500 members from government, industry, and academia, the BCserves as one of the US government s focal points for research,development, testing, evaluation, and application of Biometric -based systems. More than 60 different federal agencies and mem-bers from 80 other organizations participate in the BC cosponsors several Biometric -related projects, includ-ing some of the activities at NIST s Information TechnologyLaboratory and work at the National Biometric Test Center atSan Jose State University. The BC also cosponsors NIST sBiometrics and Smart Cards laboratory, which addresses a widerange of issues related to the interoperability, evaluation, andstandardization of Biometric technologies and smart cards, espe-cially for authentication applications like e-commerce and enter-prise-wide network September 1999, the BC held its annual conference on theconvergence of technologies for the next century.

8 The conferencehighlighted and explored new applications in e-commerce, net-work security, wireless communications, and health services. Italso addressed convergence of biometrics and related technolo-gies like smart cards and digital BC s Web site and its open listserv are two of the consor-tium s richest resources ( ).Kirk L. Kroekeris associate editor at Computer magazine. Contact him at re willing to make. If you deny access to every-one, the false-reject rate will be one and the false-alarmrate will be zero. At the other extreme, if you granteveryone access, the false-reject rate will be zero andthe false-alarm rate will be , systems operate between the two most applications, you adjust a system parameterto achieve a desired false-alarm rate, which results in acorresponding false-reject rate. The parameter settingdepends on the application. For a bank s ATM, wherethe overriding concern may be to avoid irritating legit-imate customers, the false-reject rate will be set low atthe expense of the false-alarm rate.

9 On the other hand,for systems that provide access to a secure area, thefalse-alarm rate will be the overriding system parameters can be adjusted toachieve different false-alarm rates, it often becomesdifficult to compare systems that provide performancemeasurements based on different false-alarm ,2 EVALUATION PROTOCOLSAn evaluation protocol determines how you test asystem, select the data, and measure the evaluations are administered by indepen-dent groups and tested on Biometric signatures notpreviously seen by a system. If you don t test with pre-viously unseen Biometric signatures, you re only test-ing the ability to tune a system to a particular data an evaluation to be accepted by the biometriccommunity, the details of the evaluation proceduremust be published along with the evaluation proto-col, testing procedures, performance results, and rep-resentative examples of the data set.

10 Also, theinformation on the evaluation and data should be suf-ficiently detailed so that users, developers, and ven-dors can repeat the evaluation itself should not be too hard or tooeasy. If the evaluation is too easy, performance scoreswill be near 100 percent, which makes distinguishingbetween systems nearly impossible. If the evaluationis too hard, the test will be beyond the ability of exist-ing Biometric techniques. In both cases, the results willfail to produce an accurate assessment of evaluation is just right when it spreads the per-formance scores over a range that lets you distinguishamong existing approaches and technologies. Fromthe spread in the results, the best performers can bedetermined along with the strengths and weaknessesof the technology. The strengths and weaknessesLawrence O Gorman, Veridicom the mid-1990s, optical finger-print-capture devices were bulky (aboutthe size of half a loaf of bread) and expen-sive (costing anywhere from $1,000 to$2,000).