The Bookmark Method - sagepub.com

1 10 The Bookmark MethodThe Bookmark procedure may be viewed as a logical successor to a seriesof item-mapping strategies developed in the 1990s in conjunction withstandard settings carried out for the National Assessment of EducationalProgress (NAEP) by researchers at American College Testing (ACT). Earlyitem-mapping techniques were applied less as standard-setting procedures perse than as feedback mechanisms embedded in other procedures (cf. Loomis& Bourque, 2001).In 1996, for example, researchers at ACT employed an item-mapping pro-cedure in conjunction with a Method they referred to as Mean Estimation,which was essentially an extension of the modified Angoff (1971) item-mapping procedure was applied to tests with both multiple-choiceand constructed-response items (Loomis, Bay, Yang, & Hanick, 1999). Itemmaps were used to provide feedback after a second round of item ratings forthe 1996 Science assessment and the 1998 NAEP Civics and Writing assess-ments.

2 The maps showed the location of each item in relation to the NAEP-like scale score, which was also associated with the various NAEP achievementlevel descriptors (ALDs, which are now commonly referred to as performancelevel descriptors). Each multiple-choice item was mapped in accordance withits probability of correct response for each scale score, and each constructed-response item was mapped once for each score point, that is, for the probabil-ity of obtaining a score of 1, 2, 3, or higher at each scale score techniques evolved through the course of several NAEP standard-setting studies at ACT. The Reckase chart (Reckase, 2001) wasintroduced as a way to simplify the task set before participants. With15510-Cizek (Standard) 10/23/2006 6:12 PM Page 155 Reckase charts, participants would receive their Round 2 item estimates( , the probability of a correct response by a student at the cut score formultiple-choice items and estimated raw score for constructed-responseitems for this same student or group of students), along with a preprintedtable or map of item sample Reckase chart for an individual participant is shown in Table10-1.

3 A unique Reckase chart would be developed based on each partici-pant s item ratings. The first column in the Reckase chart shown in the tablepresents scaled scores arranged from high to low. Scaled scores are used inReckase charts as a measure of overall examinee competence or ability onwhatever construct is measured by the test. Each of the remaining columnscontains information on a single item. Table 10-1 shows information onfive items with Items 1 4 being dichotomously scored multiple-choice format items and Item 5 being a constructed-response item scored on a 0 5scale. For the multiple-choice items, the data in each column show the prob-ability of an examinee at each scaled score answering that item correctly,based on the three-parameter item response model. For example, an exam-inee with an overall ability level ( , scaled score) of 170 has a .53 proba-bility of answering Item 1 correctly.

4 For constructed-response items, thevalues in a column show the expected item score for examinees at a givenscaled score location. Again considering an examinee with an ability levelof 170, the expected score of that examinee on the constructed-responseitem (Item 5) is out of Table 10-1, one value in each column appears in brackets; it is inthis way that Reckase charts are individualized for each participant. Whenused as feedback in standard setting, Reckase charts help participants gaugehow consistently they are applying their conceptualization of the minimallycompetent examinee, borderline candidate, or whatever hypothetical exam-inee is considered. The Reckase chart for a participant who is consistentlyapplying his or her conceptualization would show brackets aligned in a sin-gle row. For example, consider the participant whose judgments resultedin the values shown in the table.

5 In addition, let us assume that the partic-ipant held an implicit conceptualization that the minimally qualified exam-inee is one with an ability level (represented by a scaled score) of across the row in the table corresponding to a scaled score of 170,we see that the probability estimate ( , Angoff rating) generated by thisparticipant was .53; this participant is saying that the probability of aminimally qualified examinee answering Item 1 correctly is .53. Now, ifthis participant were applying his or her conceptualization of the mini-mally qualified examinee consistently, he or she would have generated anAngoff rating of .83 for Item 2, .34 for Item 3, and .77 for Item 4. For the156 Standard-Setting Methods10-Cizek (Standard) 10/23/2006 6:12 PM Page 156constructed-response item (Item 5), this participant would have estimatedthe minimally qualified examinee s score to be out of the Reckase chart shown in Table 10-1, however, the participantcan see that he or she is not making totally consistent judgments.

6 For theremaining three multiple-choice items (Items 2 4), the participant has esti-mated the items to be more difficult than they are for an examinee of abil-ity level 170. For example, for Item 2, the participant judged the minimallyqualified examinee to have a .57 probability of success on the item when,using the standard implied by this participant s rating of Item 2, the ratingfor Item 2 should have been .83. For the constructed-response item, thereviewer exhibited more consistent behavior with his or her implicit perfor-mance standard as shown by the fact that his or her rating of Item 5 of very close to the expected constructed-response item score of forexaminees with an overall ability level of 170. If this participant were beingperfectly consistent, the bracketed values would be aligned in a row corre-sponding to a single ability level (scaled score).Table 10-1 can be thought of as an early item map.

7 From this foundation,it was not a great step to refine the item-mapping procedure by reorderingthe items according to their difficulty. Loomis, Hanick, Bay, and Crouse(2000) reported on field trials for the 1998 NAEP Civics test in which theitem maps were reordered from least to most difficult item. These item mapsalso included brief descriptions of item content, which permitted partici-pants, at a glance, to summarize both the location and content of an itemand to reframe their own judgments of those items. From difficulty-ordereditem maps with content information and probability of correct response, theleap to an ordered test booklet with similar information was a short but sig-nificant one. Researchers at CTB/McGraw-Hill made that leap and intro-duced the Bookmark Method (Lewis, Mitzel, & Green, 1996).Overview of the Bookmark MethodThe standard Bookmark procedure (Mitzel et al.)

8 , 2001) is a complete set ofactivities designed to yield cut scores on the basis of participants reviews ofcollections of test items. The Bookmark procedure is so named because par-ticipants express their judgments by entering markers in a specially designedbooklet consisting of a set of items placed in difficulty order, with itemsordered from easiest to hardest. This booklet, called an ordered item booklet,will be described in greater detail in the next portion of this Bookmark procedure has become quite popular for several , from a practical perspective, the Method can be used for complex,The Bookmark Method 15710-Cizek (Standard) 10/23/2006 6:12 PM Page 157158 Standard-Setting MethodsTable 10-1 Example of a Reckase ChartProbabilities of Correct Response for Given Scale ScoreScale ScoreItem 1 Item 2 Item 3 Item 4 Item [.53]. [ ] [.57]. [.44] [.25]. : For multiple-choice items (Items 1 4) the values in brackets [ ] are a participant sAngoff ratings; for constructed-response items (Item 5) the value in brackets is the participant sestimated mean score for a minimally competent : Adapted from Reckase (2001).

9 10-Cizek (Standard) 10/23/2006 6:12 PM Page 158mixed-format assessments, and participants using the Method considerselected-response (SR) and constructed-response (CR) items together. Asthe prevalence of mixed-format examinations continues to increase, it islikely that the Bookmark Method will become even more widely used andthat other innovative approaches for setting performance standards in suchcontexts will be , from the perspective of those who will be asked to make judg-ments via this Method , it presents a relatively simple task to participants, andone with which, at a conceptual level, they may already be familiar. To fullygrasp the extent to which the Bookmark Method simplifies the standard-setting task, it is instructive to consider a test with four performance levels(Below Basic, Basic, Proficient, and Advanced), 60 SR items, and four CRitems (with four score points each).

10 If item-based standard-setting methodssuch as the Angoff or modified Angoff procedures were used, participantswould have 192 separate tasks to perform per round of ratings ( , threeprobability judgments for each of 64 items). With the Bookmark procedure,the same participant may still consider the content covered by the items in atest but is required to make only three judgments one for each of threebookmarks (Basic, Proficient, and Advanced) he or she will be asked toplace in a difficulty-ordered test booklet (described in more detail later inthis chapter). The task is perhaps even more streamlined because it wouldseem reasonable that the Bookmark for Advanced should be placed after thebookmark for Proficient, and that the Bookmark for Proficient should beafter the Bookmark for Basic. Thus once a participant has identified one cutscore through the placement of his or her Bookmark , it is not necessary forhim or her to start the search for the next cut score at the beginning of theordered test booklet.