Support RAND For More Information - cssip.org

1 For more InformationVisit rand at rand EuropeView document detailsSupport RANDB rowse Reports & BookstoreMake a charitable contributionLimited Electronic Distribution RightsThis document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of rand intellectual property is provided for non-commercial use only. Unauthorized posting of rand electronic documents to a non- rand website is prohibited. rand electronic documents are protected under copyright law. Permission is required from rand to reproduce, or reuse in another form, any of our research documents for commercial use.

2 For Information on reprint and linking permissions, please see rand all front matter: Jump to Page 16 The rand Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and electronic document was made available from as a public service of the rand AND FAMILIESEDUCATION AND THE ARTS ENERGY AND ENVIRONMENTHEALTH AND HEALTH CAREINFRASTRUCTURE AND TRANSPORTATION INTERNATIONAL AFFAIRSLAW AND BUSINESS NATIONAL SECURITYPOPULATION AND AGINGPUBLIC SAFETYSCIENCE AND TECHNOLOGYTERRORISM AND HOMELAND SECURITYThis product is part of the rand Corporation monograph series.

3 rand mono-graphs present major research findings that address the challenges facing the public and private sectors. All rand monographs undergo rigorous peer review to ensure high standards for research quality and Guthrie, Watu Wamae, Stephanie Diepeveen, Steven Wooding and Jonathan GrantMeasuring researchA guide to research evaluation frameworks and toolsSusan Guthrie, Watu Wamae, Stephanie Diepeveen, Steven Wooding and Jonathan GrantPrepared for the Association of American Medical CollegesMG-1217-AAMC July 2013 Measuring researchA guide to research evaluation frameworks and toolsR AND Europe is an independent.

4 Not-for-profit research organisation whose mission is to improve policy and decision making for the public good. R AND s publications do not necessarily reflect the opinions of its research clients and is a registered trademark. Copyright 2013 R AND CorporationPermission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of R AND documents to a non-R AND website is prohibited. R AND documents are protected under copyright law.

5 For Information on reprint and linking permissions, please visit the R AND permissions page ( ).Published 2013 by the R AND Corporation1776 Main Street, Box 2138, Santa Monica, CA 90407-21381200 South Hayes Street, Arlington, VA 22202-50504570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 rand URL: order R AND documents or to obtain additional Information , contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: research described in this report was prepared for the Association of American Medical Association of American Medical Colleges (AAMC) asked R AND Europe to conduct a detailed synthesis of existing and previously proposed frame-works and indicators used to evaluate research .

6 The purpose of this review is to stimulate debate among US policymakers and the leadership of US medi-cal schools and teaching hospitals on how research outcomes can be measured in different contexts and to address different needs and ultimately account to US taxpayers and others for the returns on their investment in research . The review also aims to pro-vide a guide to the key considerations that need to be taken into account when developing approaches to research evaluation, to be used as a resource by research funders, managers and report outlines the trade-offs that have to be taken into account and the contextual factors that need to be understood, drawing on experience of international approaches to research evaluation.

7 In addition, a detailed overview of six research evalua-tion frameworks of particular interest and relevance to AAMC is provided, along with a brief overview of a further eight frameworks, and discussion of the main tools used in research evaluation. The report is likely to be of interest to policymakers, research funders, institutional leaders and research managers. R AND Europe is a not-for-profit public policy research institute that aims to improve policy and decisionmaking through research and analysis. R AND Europe s clients include European govern-ments, institutions, NGOs and firms with a need for rigorous, independent, multi-disciplinary analysis.

8 This report has been peer-reviewed in accordance with R AND s quality assurance more Information about R AND Europe or this document, please contact:Dr Susan GuthrieR AND EuropeWestbrook CentreMilton RoadCambridge CB4 1 YGUnited KingdomTel. +44 (1223) 353 .. iiiTable of Figures .. viiTable of Tables .. viiiExecutive summary .. ixGlossary .. xiiChAPTEr 1 Introduction .. 1 Structure and content of this report .. 1 Frameworks and tools reviewed .. 2 ChAPTEr 2 Challenges and trade-offs in research evaluation .. 5 Characteristics of research evaluation approaches .. 5 Analysis of framework characteristics.

9 8 Analysis of tool characteristics .. 9 Stage of measurement .. 12 Approaches to common challenges .. 13 Credibility and acceptability .. 14 Challenges of 16 The future of research evaluation .. 16 ChAPTEr 3 Building a new research evaluation framework .. 19 Purpose .. 20 Characteristics of the framework .. 20 Addressing the pitfalls .. 21 Consider the context .. 21 Select the tools .. 22 Level of aggregation .. 22 Implementation .. 23 Summary: example case .. 23 Decision tree .. 24 Contentsvi Measuring research : a guide to research evaluation frameworks and toolsrEFErEnCESreference list .. 27 APPEnDICESA ppendix A Summary tables for frameworks and tools reviewed.

10 37 Appendix B Methodology .. 41 Analysis of characteristics of evaluation frameworks .. 41 Analysis of tools .. 48 Appendix C Finding out more : international examples of research evaluation frameworks .. 51 The Canadian Academy of health Sciences framework .. 53 Excellence in research for Australia .. 62 The national Institute of health research Dashboard .. 70 The research Excellence Framework .. 77 Productive Interactions .. 88 STAr METrICS .. 97 AErES .. 105 The Congressionally Directed Medical research Program .. 108 The national Commission for University Evaluation and Accreditation.