Support RAND For More Information - cssip.org

1 For more InformationVisit rand at rand EuropeView document detailsSupport RANDB rowse Reports & BookstoreMake a charitable contributionLimited Electronic Distribution RightsThis document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of rand intellectual property is provided for non-commercial use only. Unauthorized posting of rand electronic documents to a non- rand website is prohibited. rand electronic documents are protected under copyright law. Permission is required from rand to reproduce, or reuse in another form, any of our research documents for commercial use. For Information on reprint and linking permissions, please see rand all front matter: Jump to Page 16 The rand Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and electronic document was made available from as a public service of the rand AND FAMILIESEDUCATION AND THE ARTS ENERGY AND ENVIRONMENTHEALTH AND HEALTH CAREINFRASTRUCTURE AND TRANSPORTATION INTERNATIONAL AFFAIRSLAW AND BUSINESS NATIONAL SECURITYPOPULATION AND AGINGPUBLIC SAFETYSCIENCE AND TECHNOLOGYTERRORISM AND HOMELAND SECURITYThis product is part of the rand Corporation monograph series.

2 rand mono-graphs present major research findings that address the challenges facing the public and private sectors. All rand monographs undergo rigorous peer review to ensure high standards for research quality and Guthrie, Watu Wamae, Stephanie Diepeveen, Steven Wooding and Jonathan GrantMeasuring researchA guide to research evaluation frameworks and toolsSusan Guthrie, Watu Wamae, Stephanie Diepeveen, Steven Wooding and Jonathan GrantPrepared for the Association of American Medical CollegesMG-1217-AAMC July 2013 Measuring researchA guide to research evaluation frameworks and toolsR AND Europe is an independent, not-for-profit research organisation whose mission is to improve policy and decision making for the public good. R AND s publications do not necessarily reflect the opinions of its research clients and is a registered trademark.

3 Copyright 2013 R AND CorporationPermission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of R AND documents to a non-R AND website is prohibited. R AND documents are protected under copyright law. For Information on reprint and linking permissions, please visit the R AND permissions page ( ).Published 2013 by the R AND Corporation1776 Main Street, Box 2138, Santa Monica, CA 90407-21381200 South Hayes Street, Arlington, VA 22202-50504570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 rand URL: order R AND documents or to obtain additional Information , contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: research described in this report was prepared for the Association of American Medical Association of American Medical Colleges (AAMC) asked R AND Europe to conduct a detailed synthesis of existing and previously proposed frame-works and indicators used to evaluate research.

4 The purpose of this review is to stimulate debate among US policymakers and the leadership of US medi-cal schools and teaching hospitals on how research outcomes can be measured in different contexts and to address different needs and ultimately account to US taxpayers and others for the returns on their investment in research. The review also aims to pro-vide a guide to the key considerations that need to be taken into account when developing approaches to research evaluation, to be used as a resource by research funders, managers and report outlines the trade-offs that have to be taken into account and the contextual factors that need to be understood, drawing on experience of international approaches to research evaluation.

5 In addition, a detailed overview of six research evalua-tion frameworks of particular interest and relevance to AAMC is provided, along with a brief overview of a further eight frameworks, and discussion of the main tools used in research evaluation. The report is likely to be of interest to policymakers, research funders, institutional leaders and research managers. R AND Europe is a not-for-profit public policy research institute that aims to improve policy and decisionmaking through research and analysis. R AND Europe s clients include European govern-ments, institutions, NGOs and firms with a need for rigorous, independent, multi-disciplinary analysis. This report has been peer-reviewed in accordance with R AND s quality assurance more Information about R AND Europe or this document, please contact:Dr Susan GuthrieR AND EuropeWestbrook CentreMilton RoadCambridge CB4 1 YGUnited KingdomTel.

6 +44 (1223) 353 .. iiiTable of Figures .. viiTable of Tables .. viiiExecutive summary .. ixGlossary .. xiiChAPTEr 1 Introduction .. 1 Structure and content of this report .. 1 Frameworks and tools reviewed .. 2 ChAPTEr 2 Challenges and trade-offs in research evaluation .. 5 Characteristics of research evaluation approaches .. 5 Analysis of framework characteristics .. 8 Analysis of tool characteristics .. 9 Stage of measurement .. 12 Approaches to common challenges .. 13 Credibility and acceptability .. 14 Challenges of 16 The future of research evaluation .. 16 ChAPTEr 3 Building a new research evaluation framework .. 19 Purpose .. 20 Characteristics of the framework .. 20 Addressing the pitfalls .. 21 Consider the context .. 21 Select the tools .. 22 Level of aggregation.

7 22 Implementation .. 23 Summary: example case .. 23 Decision tree .. 24 Contentsvi Measuring research: a guide to research evaluation frameworks and toolsrEFErEnCESreference list .. 27 APPEnDICESA ppendix A Summary tables for frameworks and tools reviewed .. 37 Appendix B Methodology .. 41 Analysis of characteristics of evaluation frameworks .. 41 Analysis of tools .. 48 Appendix C Finding out more : international examples of research evaluation frameworks .. 51 The Canadian Academy of health Sciences framework .. 53 Excellence in research for Australia .. 62 The national Institute of health research Dashboard .. 70 The research Excellence Framework .. 77 Productive Interactions .. 88 STAr METrICS .. 97 AErES .. 105 The Congressionally Directed Medical research Program.

8 108 The national Commission for University Evaluation and Accreditation .. 110 The national Institution for Academic Degrees and University Evaluation .. 112 The national research Foundation Knowledge Management and Evaluation Directorate .. 114 The Performance Based research Fund .. 116 The Spanish State Programme for University Quality Assessment .. 118 The Standard Evaluation Protocol .. 120 Appendix D Finding out more : the tools available for research evaluation .. 123 Bibliometrics .. 125 Case studies in research evaluation .. 130 Data mining for research evaluation .. 134 Document review for research evaluation .. 143 Economic analysis .. 144 Logic models .. 148 Peer review for research evaluation .. 154 Site visits for research evaluation.

9 159 Survey instruments for research evaluation .. 161 Visualisation .. 169 Appendix E Information on other reports and frameworks reviewed and consulted .. 173 Figure Comparing characteristics and purposes of research evaluation frameworks .. 9 Figure Mapping of tools against Group 1 and Group 2 characteristics .. 10 Figure Unit of aggregation used in six frameworks investigated in detail .. 12 Figure Measures used in six research evaluation frameworks .. 13 Figure A decision tree for developing a research evaluation framework .. 25 Figure A1 Scatter plot matrix of the correlations between the characteristics of frameworks .. 45 Figure A2 Scree plot to determine the number of factors used in the analysis of the characteristics of frameworks.

10 47 Figure A3 Plot showing the alignment of the six frameworks investigated in detail with the five most significant factors .. 49 Figure A4 CAHS Logic Model Framework built on a payback logic model .. 54 Figure A5 The CAHS framework .. 58 Figure A6 Impact categories cross mapped for the biomedical research pillar to create a specialised framework .. 59 Figure A7 A SWOT analysis for the CAHS framework .. 61 Figure A8 Example of aggregated output from ER A 2010: data and rating summary .. 66 Figure A9 Example of aggregated output from ER A 2010: contribution to national landscape (%) .. 67 Fig ure A10 A SWOT analysis for ER A .. 69 Fig ure A11 Template for collecting financial data for the NIHR Dashboard framework .. 72 Figure A12 The integrated NIHR Dashboard structure.