Descriptive analysis in education: A guide for researchers

1 March 2017 Descriptive analysis in education : a guide for researchers Susanna Loeb Pamela Morris Stanford University New York University Susan Dynarski Sean Reardon University of Michigan Stanford University Daniel McFarland Sarah Reber Stanford University UCLA Key Themes audience. Descriptive analysis characterizes the world or a phenomenon answering questions about who, what, where, when, and to what extent. Whether the goal is to identify and describe trends and variation in populations, create new measures of key phenomena, or describe samples in studies aimed at identifying causal effects, description plays a critical role in the scientific pro- cess in general and education research in particular.

2 Descriptive analysis stands on its own as a research product, such as when it identifies socially important phenomena that have not previously been rec- ognized. In many instances, description can also point toward causal under- standing and to the mechanisms behind causal relationships. No matter how significant a researcher s findings might be, they contribute to knowledge and practice only when others read and understand the conclusions. Part of the researcher s job and expertise is to use appropriate analytical, communication, and data visualization methods to translate raw data into reported findings in a format that is useful for each intended Department of education Betsy DeVos, Secretary Institute of education Sciences Thomas W.

3 Brock, Commissioner for education Research Delegated the Duties of Director National Center for education Evaluation and Regional Assistance Audrey Pendleton, Acting Commissioner Elizabeth Eisner, Acting Associate C ommissioner Amy Johnson, Project Officer NCEE 2017 4023 The National Center for education Evaluation and Regional Assistance (NCEE) conducts unbiased large-scale evaluations of education programs and practices supported by federal funds; provides research-based technical assistance to educators and policymakers; and sup-ports the synthesis and the widespread dissemination of the results of research and evaluation throughout the United States.

4 March 2017 This report was prepared for the Institute of education Sciences (IES) by Decision Information Resources, Inc. under Contract ED-IES-12-C-0057, Analytic Technical Assistance and Development. The content of the publication does not necessarily reflect the views or policies of IES or the Department of education nor does mention of trade names, commercial products, or organizations imply endorsement by the Government. This report is in the public domain. While permission to reprint this publication is not necessary, it should be cited as: Loeb, S.

5 , Dynarski, S., McFarland, D., Morris, P., Reardon, S., & Reber, S. (2017). Descriptive anal- ysis in education : a guide for researchers . (NCEE 2017 4023). Washington, DC: Department of education , Institute of education Sciences, National Center for education Evaluation and Re- gional Assistance. This report is available on the Institute of education Sciences website at The authors would like to thank Tom Szuba, of Quality Information Partners, for his substantial role as a contributing writer.

6 Ii Contents About This Document v Purpose v Why Now? v Intended Audience v Organization vi Chapter 1. Why Should Anyone Care about Descriptive analysis ? 1 Descriptive analysis and the Scientific Method 2 Descriptive An alysis as Stand- Alone Resear c h 2 Descriptive An alysis as a Com p onent of C a us al Rese arch 4 The researcher s Role 6 Chapter 2.

7 Approaching Descriptive analysis 8 Approaching Descriptive analysis as an Iterative Process 8 Meaningful Descriptive analysis Reveals Socially Important Patterns 9 E x amples of Descriptiv e Studies T h at Reveal Consequential Phenomena 10 Descriptive analysis to Support Causal Understanding 12 Planning an In tervention Strategy 13 T a rg e t i n g I n te rv e n ti on s 13 Contributin g t o the Interpret a tion of Caus

8 Al Study 14 Assessing Variation in T r eatment Impact 15 Prioritizing Potential Causal Mediators 16

9 Approaching Descriptive analysis : Summary 17 Chapter 3. Conducting Descriptive analysis 18 Key Terminology and Methodological Considerations 18 Res e ar ch Q ues t i ons 18 Constructs 19 Measures 20 Samples 22 Using Data to Answer Research Questions 22 Statistical Adjustments 23 Comparisons 24 Groupings, Networks, and Clusters 25 Cautions Regarding Uncertainty and Fishing 26 Conducting Descriptive analysis : Summary 27 Chapter 4. Communicating Descriptive analysis 28 Communicating Data Visually 28 The Process of Communicating the Message 29 How to Fr ame Visualization N eeds 29 Common

10 Approaches to Data Visualization 31 Tables 32 Graphs 33 Communicating Descriptive analysis : Summary 37 Chapter 5. Summary and Conclusions 39 Appendix A. Resources Related Especially to Communications and Visualization 1 Appendix B. References B-1 iii ivBoxes Box 1. Descriptive analysis Is a Critical Component of Research 2 Box 2. Examples of Using Descriptive Analyses to Diagnose Need and Target Intervention on the Topic of Summer Melt 3 Box 3. An Example of Using Descriptive analysis to Evaluate Plausible Causes and Generate Hypotheses 4 Box 4. An Example of Using Descriptive analysis to Interpret Causal Research 5 Box 5.