Statistical Analysis Of Multiple Choice Testing

1 AU/ACSC/042/2001-04 AIR COMMAND AND STAFF COLLEGE AIR UNIVERSITY Statistical Analysis OF Multiple Choice Testing by Mark A. Colbert, Major, USAF A Research Report Submitted to the Faculty In Partial Fulfillment of the Graduation Requirements Advisor: Lieutenant Colonel Thomas P. Himes, Jr. Maxwell Air Force Base, Alabama April 2001 Report Documentation PageReport Date 01 APR2001 Report Type N/ADates Covered ( to) - Title and Subtitle Statistical Analysis of Multiple Choice TestingContract Number Grant Number Program Element Number Author(s) Colbert, Mark Number Task Number Work Unit Number Performing Organization Name(s) and Address(es) Air Command and Staff College Air University MaxwellAFB, ALPerforming Organization Report Number Sponsoring/Monitoring Agency Name(s) and Address(es) Sponsor/Monitor s Acronym(s) Sponsor/Monitor s Report Number(s)

2 Distribution/Availability Statement Approved for public release, distribution unlimitedSupplementary Notes Abstract Subject Terms Report Classification unclassifiedClassification of this page unclassifiedClassification of Abstract unclassified Limitation of Abstract UUNumber of Pages 56 Disclaimer The views expressed in thi s academic research paper are those of the author and do not reflect the official policy or position of the US government or the Department of Defense. In accordance with Air Force Instruction 51-303, it is not copyrighted, but is the property of the United States government. ii Contents Page DISCLAIMER.

3 Ii ILLUSTRATIONS ..v vi vii BACKGROUND ..1 Introduction and Problem Scope of Analysis ..2 HISTORICAL REVIEW ..3 Why Use Statistics for Test Item Analysis ..3 Ease Differentiation Index ..4 Correlation Coefficients ..5 Quantitative Qualitative ACSC Distance Learning Department s Current Methods ..10 TAD Software Program ..13 Inputs and Output User Friendliness ..15 ITEMAN Software Inputs and Output User Friendliness ..17 Quantitave Qualitative Analysis ..19 ACSC Distance Learning Department s Current Methods Analysis ..20 iii Summary of Findings ..24 ACSC Distance Learning Department s Current Methods.

4 24 TAD versus ITEMAN ..25 Recommendations ..26 ITEMAN as the Preferred Quantitative Measurements ..26 Qualitative Guidelines ..27 SAMPLE OUTPUTS FROM SOFTWARE TAD Sample Output Using EI, DI, and Point Biserial ITEMAN Sample Output Using EI, DI, and Point Biserial ITEMAN Sample Output Using EI, DI, and Biserial Correlations ..35 QUESTION WRITING GUIDELINES ..40 Maxwell Academic Instructor School Test Item Analysis Handout s section on Qualitative James D. Hansen and Lee Dexter s Item-writing GLOSSARY ..44 iv Illustrations Page Figure 1 Unit Normal Curve and Values of q, p and y ..8 Figure 2 TAD Item Statistics window sample Figure 3 ITEMAN Sample Output Using EI, DI and Biserial Coefficients.

5 16 Figure 4 TAD Sample Output using EI, DI and Point Biserial (Part 1of 2)..28 Figure 5 TAD Sample Output using EI, DI and Point Biserial (Part 2 of 2)..29 Figure 6 ITEMAN Sample Output Using EI, DI and Point Biserial (Part 1 of 5) ..30 Figure 7 ITEMAN Sample Output Using EI, DI and Point Biserial (Part 2 of 5) ..31 Figure 8 ITEMAN Sample Output Using EI, DI and Point Biserial (Part 3 of 5) ..32 Figure 9 ITEMAN Sample Output Using EI, DI and Point Biserial (Part 4 of 5) ..33 Figure 10 ITEMAN Sample Output Using EI, DI and Point Biserial (Part 5 of 5) ..34 Figure 11 ITEMAN Sample Output Using EI, DI and Biserial (Part 1 of 5) ..35 Figure 12 ITEMAN Sample Output Using EI, DI and Biserial (Part 2 of 5).

6 36 Figure 13 ITEMAN Sample Output Using EI, DI and Biserial (Part 3 of 5) ..37 Figure 14 ITEMAN Sample Output Using EI, DI and Biserial (Part 4 of 5) ..38 Figure 15 ITEMAN Sample Output Using EI, DI and Biserial (Part 5 of 5) ..39 v Preface The United States Air Force Air Command and Staff Col lege s Distance Learning Department offered this research topic as an opportunity to evaluate their process of analyzing Multiple - Choice questions used in their tests. I selected this topic because I have an interest in the topic and have a bachelor s degree in mathematics. Like many people who have taken Multiple - Choice question tests, I have always wondered how test givers decide to throw out bad questions.

7 This research paper was an excellent opportunity to explore this question in depth. I would like to acknowledge Lieutenant Colonel Thomas Himes, my research advisor, for his support and background knowledge on the subject. I would like to thank Dr. Thomas R. Renckly and Mr. Michael Zieky for allowing me to interview them at length to gain from their expert knowledge in the test item Analysis and evaluation field. I would also like to thank many others too numerous to name who have helped me complete this research effor t. AU/ACSC/042/2001-04 Abstract The Multiple - Choice question exam is a very popular method of evaluation used by educators everywhere.

8 The Air Command and Staff College Distance Learning Department uses Multiple - Choice exams for Testing non-residence students. ACSC cur rently uses the Test Analysis and Development (TAD) software program s two quantitative measurements, Ease Index and Differentiation Index, to flag possible problem questions for qualitative review. They also use student feedback to flag questions for review. ACSC uses the Maxwell Academic Instructor School s Test Item Analysis Handout to examine qualitatively the flagged questions to determine which need revision. The purpose of this paper is to determine if the ACSC Distance Learning Department is doing a good job at test evaluation and whether there are better ways to determine the quality, effectivene ss and fairness of Multiple - Choice questions.

9 This paper compares the TAD program to the ITEMAN progr am for quantitative ana lysis. For qualitative Analysis , prior studies and guidelines are compared to those used by ACSC. This paper found that ACSC is doing a good job at test evaluation by using the best threshold values for the Ease Index and Differentiation Index to flag items. This paper recommends that ACSC use the ITEMAN software program because of its ease, speed and superior output. This paper recommends that ACSC use the Biserial Correlation Coefficient as well to flag questions. Lastly, this paper recommends that ACSC use Hansen and Dexter s Item-writing Guidelines for qualitative review of flagged questions.

10 Vii Chapter 1 Background Introduction and Problem Definition The Multiple - Choice question test is perhaps the most popular educational evaluation method used at all levels. The challenge of using this method is designing well-written questions that are reliable and can discriminate the more knowledgeable students from the less knowledgeable students. Every question can be eva luated qualitatively (well written) and quantitatively (reliable and able to discriminate). Qualitative methods can help determine if a question is poorly written. Quantitatively, computer programs provide Statistical measures to help determine if a question did not statistically perform well.