Test Reliability—Basic Concepts

1 Test reliability Basic Concepts Samuel A. LivingstonJanuary 2018 Research Memorandum ETS RM 18-01 ETS Research Memorandum SeriesEIGNOR EXECUTIVE EDITORJ ames CarlsonPrincipal PsychometricianASSOCIATE EDITORSB eata Beigman KlebanovSenior Research ScientistHeather BuzickSenior Research ScientistBrent BridgemanDistinguished Presidential AppointeeKeelan EvaniniResearch DirectorMarna Golub-SmithPrincipal PsychometricianShelby HabermanDistinguished Research Scientist, EdusoftAnastassia LoukinaResearch ScientistJohn MazzeoDistinguished Presidential AppointeeDonald PowersPrincipal Research ScientistGautam PuhanPrincipal PsychometricianJohn SabatiniManaging Principal Research ScientistElizabeth StoneResearch ScientistRebecca ZwickDistinguished Presidential AppointeePRODUCTION EDITORSKim FryerManager, Editing ServicesAyleen GontzSenior EditorSince its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields.

2 In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Memorandum series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Memorandum series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Daniel Eignor Editorship is named in honor of Dr.

3 Daniel R. Eignor, who from 2001 until 2011 served the Research and Development division as Editor for the ETS Research Report series. The Eignor Editorship has been created to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at reliability Basic ConceptsSamuel A. LivingstonEducational Testing Service, Princeton, New JerseyJanuary 2018 Corresponding author: S. A. Livingston, E-mail: citation: Livingston, S. A. (2018). Test reliability Basic Concepts (Research Memorandum No. RM-18-01). Princeton, NJ: Educational Testing other ETS-published reports by searching the ETS ReSEARCHER database at obtain a copy of an ETS research report, please visit Editor: Gautam PuhanReviewers: Shelby Haberman and Marna Golub-SmithCopyright 2018 by Educational Testing Service.

4 All rights , the ETS logo, GRE, MEASURING THE POWER OF LEARNING, and TOEFL are registered trademarks of Educational Testing Service (ETS). All other trademarks are the property of their respective owners. RM-18-01 i Abstract The reliability of test scores is the extent to which they are consistent across different occasions of testing, different editions of the test, or different raters scoring the test taker s responses. This guide explains the meaning of several terms associated with the concept of test reliability : true score, error of measurement, alternate-forms reliability , interrater reliability , internal consistency, reliability coefficient, standard error of measurement, classification consistency, and classification accuracy.

5 It also explains the relationship between the number of questions, problems, or tasks in the test and the reliability of the scores. Key words: reliability , true score, error of measurement, alternate-forms reliability , interrater reliability , internal consistency, reliability coefficient, standard error of measurement, classification consistency, classification accuracy RM-18-01 ii Preface This guide grew out of a class that I teach for staff at Educational Testing Service (ETS). The class is a nonmathematical introduction to the topic, emphasizing conceptual understanding and practical applications. The class consists of illustrated lectures, interspersed with written exercises for the participants.

6 I have included the exercises in this guide, at roughly the same points as they occur in the class. The answers are in the appendix at the end of the guide. In preparing this guide, I have tried to capture as much as possible of the conversational style of the class. I have used the word we to refer to myself and most of my colleagues in the testing profession. (We tend to agree on most of the topics discussed in this guide, and I think it will be clear where we do not.) RM-18-01 iii Table of Contents Instructional Objectives .. 1 Prerequisite Knowledge .. 2 What Factors Influence a Test Score? .. 2 The Luck of the Draw .. 3 Reducing the Influence of Chance Factors.

7 4 Exercise: Test Scores and Chance .. 5 What Is reliability ? .. 6 reliability Is Consistency .. 6 reliability and Validity .. 7 Exercise: reliability and Validity .. 8 Consistency of What Information? .. 8 True Score and Error of Measurement .. 9 reliability and Measurement Error .. 11 Exercise: Measurement Error .. 11 reliability and 12 Alternate-Forms reliability and Internal Consistency .. 13 Interrater reliability .. 15 Test Length and reliability .. 16 Exercise: Interrater reliability and Alternate-Forms reliability .. 17 reliability and Precision .. 18 reliability Statistics .. 19 The reliability Coefficient .. 19 The Standard Error of Measurement .. 20 How Are the reliability Coefficient and the Standard Error of Measurement Related?

8 22 Test Length and Alternate-Forms reliability .. 22 Number of Raters and Interrater reliability .. 24 reliability of Differences Between Scores .. 25 Demystifying the Standard Error of Measurement .. 26 Exercise: The reliability Coefficient and the Standard Error of Measurement .. 26 reliability of Essay 27 RM-18-01 iv reliability of Classifications and Decisions .. 28 Summary .. 30 32 Appendix. Answers to Exercises .. 32 Exercise: Test Scores and Chance .. 32 Exercise: reliability and Validity .. 33 Exercise: Measurement Error .. 33 Exercise: Interrater reliability and Alternate-Forms reliability .. 35 Exercise: The reliability Coefficient and the Standard Error of Measurement.

9 35 Notes .. 37 S. A. Livingston Test reliability Basic Concepts RM-18-01 1 Instructional Objectives Here is a list of things I hope you will be able to do after you have read this guide and done the written exercises: List three important ways in which chance can affect a test taker s score and some things that test makers can do to reduce these effects. Give a brief, correct explanation of the concept of test reliability . Explain the difference between reliability and validity and how these two Concepts are related. Explain the meaning of the terms true score and error of measurement and why it is wise to avoid using these terms to communicate with people outside the testing profession.

10 Give an example of an unwanted effect on test scores that is not considered error of measurement. Explain what alternate-forms reliability is and why it is important. Explain what interrater reliability is, why it is important, and how it is related to alternate-forms reliability . Explain what internal consistency is, why it is often used to estimate reliability , and when it is likely to be a poor estimate. Explain what the reliability coefficient is, what it measures, and what additional information is necessary to make it meaningful. Explain what the standard error of measurement is, what it measures, and what additional information is necessary to make it meaningful.