Measurement Types of Instruments

1 1 Measurement - Types of Instruments - 1rsmichaelMeasurementTypes of InstrumentsY520 Strategies for Educational InquiryRobert S MichaelMeasurement - Types of Instruments - 2rsmichaelFirst Questions What is the stated purpose of the instrument? Is it to: Collect information? Predict a specified performance ( , The purposeof the SAT is to predict first college semester GPA). Measure achievement? Measure personality? Measure attitude toward [an attitudinal object]? The adequacy of a test can be judged only if you knowits stated - Types of Instruments - 3rsmichaelBackground Terms & Ideas Instruments : Collections of items or questions intendedto measure, collectively (total score) levels oftheoretical variables (constructs) not easily observabledirectly.

2 Examples: intelligence optimism quality of life job satisfaction depression anxiety political orientationMeasurement - Types of Instruments - 4rsmichaelInstruments vs Indices Instrument(aka, test, scale): Type of response toitems on an instrument are presumed to be caused bythe underlying construct you are trying to measure. Index:In contrast, an index consists of items thatdefinethe construct. Instrument example: Given a scale that measuresoptimism, we believe that more positive responses( , higher total scores) are caused by someone whohas higher levels of - Types of Instruments - 5rsmichaelInstruments vs Indices Index example: A socioeconomic status (SES) indexmight include a Measurement of education level.

3 Buthaving more education is not caused by higher SES;instead having more education influences SES. An SES index measures SES, by definition. Noattempt is made to measure a latent - Types of Instruments - 6rsmichaelThree Types of Instruments Questionnaires. Goal: Collection of information. Noattempt to measure underlying latent construct. Teacher Turnover Survey Early Literacy Intervention Grant Prog. Survey Cognitive domaininstruments. Goal: Measureachievement or aptitude. Affective domaininstruments. Goal: Measurepersonality constructs, attitudes, beliefs, interests, - Types of Instruments - 7rsmichaelQuestionnaire Development Guidelines Clear, concise, unambiguous statements.

4 Be aware of respondent reading levels. Pilot test, pilot test, pilot test. Frary, Robert (nd). A brief guide to questionnairedevelopment. [Retrieve ] Dillman, D. A. (1978).Mail and telephone surveys: Thetotal design method. New York: John - Types of Instruments - 8rsmichaelExample of Wording When presented with difficult professional situationswhere a superior censures you for an act for whichyou are not responsible, how frequently do yourespond in an assertive way? Allofthetime Some of the time None of the time Notice the vocabulary level and sentence - Types of Instruments - 9rsmichaelAlternate Wording When your boss blames you for something you didnot do, how often do you stick up for yourself?

5 Allofthetime Some of the time None of the time Which wording is preferable? Why? Measurement - Types of Instruments - 10rsmichaelExample 2: Alternate Wording During the past 4 weeks, how often have you felt asense of loneliness? Allofthetime Sometimes Fromtimetotime Never6 Measurement - Types of Instruments - 11rsmichaelExample 2: Alternate Wording How often in the past month have you felt alone inthe world? Every day Somedays Occasionally Never Is this an improvement? Might some problems stillexist? Measurement - Types of Instruments - 12rsmichaelSteps in Constructing an Instrument Determine clearly what it is you want to measure.

6 Generate a pool of possible items (about 60 for a 20 item instrument). Decide on question format (yes/no, strongly agree, agree, etc) Decide how to score the instrument ( on an optimism scale, allresponses that indicate optimism should get higher scores, so that themore optimistic you are, the higher your total score). Administer the instrument to a pilot sample of people: 100 - 300. Scale purification: Evaluate each item using the pilot data: The goal is to select the smallest possible subset of items that is internally reduces respondent burden (few questions) and high reliability (internalconsistency).

7 Use Reliability procedure in Spss. Calculate Cronbach s alpha for various subsets of items and calculate the item-remainder coefficients (the correlation of an item with the sum of the remaining items). Drop items that lower Cronbach s alpha If alpha remains low, use Spearman Brown Prophecy formula to tell you how many moreitems you - Types of Instruments - 13rsmichaelConstructing Instruments Preceding steps work well for questionnaires, attitudemeasures, personality scales. Some what different procedures are needed forcognitive domain - Types of Instruments - 14rsmichaelCognitive Domain Instruments Achievement Instruments Performance limited to specific domain ( , music) Classroom Tests (teacher developed) High stakes tests Iowa Test of Basic Skills ISTEP Aptitude Instruments Performance generalized 8 Measurement - Types of Instruments - 15rsmichaelAchievement Tests Classroom tests: measure student achievement inspelling, math, English, history, science, etc.

8 The latent construct is achievement The operational definition is number correct on aparticular test. Items on a test are a sample from all possiblequestions that could be asked from a particularacademic domain. For test results to be interpretable, test (or subscale)should include items from only one - Types of Instruments - 16rsmichaelAchievement Tests (continued) Classroom tests (continued): If a test contained bothspelling and math items and number correct weresummed, what would the score mean? Instead, sum the correct spelling items separately fromthe correct math items.

9 Achievement tests can be either norm-based orcriterion-based. Can be objectively scored items (true/false, multiplechoice, matching, ordering) or performanceassessments (aka, essay), or authentic assessments9 Measurement - Types of Instruments - 17rsmichaelAchievement Test Items Highly sophisticated guidelines for item writing [enter reference here] Items differ in the level of cognitive functioningrequired to answer them: Knowledge Comprehension Application Analysis Synthesis/EvaluationMeasurement - Types of Instruments - 18rsmichaelBloom s Taxonomy: Test Items Bloom s Taxonomy describes different levels ofcognitive functioning.

10 Items can be classifieddepending on the level of cognitive functioningrequired to answer. Objectively scored items ( , multiple choice,true/false, matching, short answer) can be written torequire either knowledge, comprehension, application,or analysis. Carefully constructed essay questions are used torequire synthesis/evaluation - Types of Instruments - 19rsmichaelBloom s Taxonomy: Levels of Knowledge Knowledge:Recognize or recall key facts, definitions,concepts, rules, principles. ( lowest level ) Key words: Define, identify, label, list, name Comprehension:Understanding of abstract or explain similarities and differences.