Using the binomial distribution to assess effort: …

1 253 Using the binomial distribution to assesseffort: forced - choice testing inneuropsychological settingsFaulder Colby Oregon Health Sciences University, Portland, OR,USAThe binomial distribution is often, but prematurely, rejectedas a tool for assessing effort. This study extended previ-ous research Using published clinical and computer-generatedpseudo subject data for theTest of Memory Malingering(TOMM). The efficiencies of eight cut points based upon in-verse binomial distribution functions were compared with thecut point recommended in the test manual for making correctclassifications, and a new statistic, the total number of errors,was also compared with the test manual cut point. Repeatedmeasures, multivariate, and univariate ANOVAs, Bonferroni-corrected post-hoc t-tests, and normal curve density functionswere employed to assess the homogeneity of groups withinexperimental conditions.

2 Based upon these analyses, changeswere recommended in the decision rules for theTOMM, andstrategies for improving the norms for theTOMMand forneuropsychological assessment instruments, generally, : Malingering, effort, motivation, neuropsycholog-ical, assessment, TOMM, binomial1. IntroductionThe February 2000 issue of theAmerican Psychol-ogistcontained several articles devoted to the impactafter 50 years of the 1949 conference in Boulder, CO,on graduate education in clinical psychology. One pri-mary goal of academic research recommended at thatconference was to Improve the accuracy and reliabilityof diagnostic procedures [1]. From its inception in the1940 s, neuropsychology in the United States often has Address for correspondence: F.

3 Colby, PhD, 520 SW 6th Avenue,Suite 950, Portland, OR 97204-1513, USA. Tel. & Fax: +1 503 2485643 or +1 800 562 9742; E-mail: upon statistical approaches, and diagnoses haveoften relied heavily upon actuarial methods. Parallelwith this movement, in Russia, was the emergence ofa more clinical approach based upon the writings andteachings of Luria [2].As the use of neuropsychological expertise in foren-sic contexts has expanded, concerns have been raisedthat litigants might feign or exaggerate impairments [3].Therefore, effort assessment has received much at-tention by clinicians and researchers [4]. In the late1980 s, Binder and Pancratz [5] reported findings abouta forced - choice visual memory task Using colored pen-cils to assess factitious visual memory and Mandelcorn [6] used a similar procedureto assess feigned study of effort assessment in neuropsychologyhas grown so rapidly that it is now fairly common inprofessional journals to read about new tests or tech-niques [7 13] and about the validity of effort assess -ment in forensic settings [14 16].

4 Books are devotedto the topics of malingering and deception [17,18], anddetection of deception during interviews has also beenthe focus of much research [19,20].Generally, actuarial effort assessment tools are oftwo types [3]: procedures which are derived from testsor techniques which are designed to assess real cogni-tive deficits [10,21,22], and techniques which are de-signed purely to assess effort [3,7,13]. A potentialproblem with procedures which have been derived fromability measures is that very impaired persons may failthem [10,15], resulting in high false positive error rates(FPR). However, if the sensitivity of an assessmenttechnique is reduced to guard against high FPR, thenhigh false negative rates (FNR) may occur [23].

5 In the-ory, an advantage of instruments designed to assess onlyeffort is that even very damaged persons giving theirbest efforts should pass them, while persons not givingtheir best efforts, unaware of how impaired persons re-ally perform on them, should fail them. Well-designedNeuroRehabilitation 16 (2001) 253 265 ISSN 1053-8135 / $ 2001, IOS Press. All rights reserved254F. Colby / Using the binomial distribution to assess effortforced- choice tests may provide such tools [7]. In thetypical forced - choice effort test, a target stimulus ispresented, followed by a distraction task [7], a waitingperiod [24], or more test stimuli [13], and then recog-nition memory is tested Using one or more foils as dis-tractors.

6 Decisions about effort are usually based uponcut scores with empirically determined hit tests like these and articles about them havementioned the binomial theorem or probability dis-tribution [3,25,26], with perhaps one exception [27],the assumption has been stated or implied that theprobability, p, of a correct answer on a two stimulus, forced - choice test is equal to under pure chance conditions [7,11,25,26], even though, as one reviewerpointed out, the cognitively impaired persons on whomthese tests have been normed have not generally beenassumed to be suffering from a no memory condi-tion. Since very few persons, impaired or not, missas many as half the items [7], Using a decision ruleof poorer than chance , where chance is assumed tomean thatp= , would naturally result in a highFNR.

7 Under this set of assumptions, the use of the bi-nomial frequency distribution has generally been re-jected in favor of the use of cut scores developed em-pirically, but without reference to underlying frequencydistributions. Some authors have even stated that highcut score decision rules have been used because theforced- choice effort test was necessarily an insensitiveprocedure [28].This widespread rejection of the binomial distribu-tion is apparently based upon the erroneous notion thatunder the binomial distribution , the probability,p,ofacorrect response on a two stimulus forced - choice testmust alwaysbe 50% under pure chance , statistics textbooks universally refer to thevariables, p and q, not to constants, when discussing thebinomial distribution .

8 Hays [23] writes, Thus, whatwe are calling the binomial distribution is actually awhole family of binomial probability binomial distribution usually refers to the familyof distributions having the same rule, and a binomialdistribution is a particular one of this family found byfixingNandp[italics the author s] (p. 120). That this elementary understanding of the binomialdistribution should either be unknown to, or ignored by,so many researchers in neuropsychology is this fundamental error, Colby [29] argued thatthe binomial distribution had been prematurely rejectedas a tool for assessing effort with forced - choice the probability,p, of a correct choice under a purechance condition had been shown to be about 50%for cognitively impaired persons during the norming ofeffort tests, then continuing to assume thatp= be prudent.

9 However, since that has repeatedlynot been the case, it should by now be obvious that theexpected values ofpfor impaired persons are greaterthan 50% for these types of tests, with the exact valuesofpbeing dependent upon the populations upon whichthe tests have been any test where there is only one correct answerfor each test item, the underlying theoretical probabil-ity distribution for the total test score will be binomialif (1) the items are independent and (2) the outcomesfor each item are random events [23]. This is true evenif the probabilities of correct answers to test items arenot equal [27]. So long as there are only two outcomesper item (right and wrong), test items are independent,and no extraneous factors influence responses, the dis-tribution of the total test scores across multiple admin-istrations of the test will fit a binomial probability dis-tribution withNbeing equal to the number of items onthe test andpbeing equal to the long-run average per-cent test score (C.)

10 Gullion, personal communication,6/7/2000).Perhaps it is the use of the terms, random and inde-pendent, that has been the source of confusion about thebinomial distribution . Random implies only that thereare no extraneous influences to affect responses to testitems. Using a bag of 90 red and 10 blue marbles as ametaphor, each draw is random if its outcome dependssolely upon the physical act of pulling a marble fromthe bag, even if the chances are nine to one that the out-come of any particular draw will be red. One exampleof an extraneous influence in effort testing would be themotivation to get fewer right answers than the personis capable of getting when giving their best and honesteffort. Independent means that the outcomes on itemsare unrelated.