APPLYING QUALITATIVE EVALUATION METHODS

1 165 CHAPTER5 APPLYING QUALITATIVEEVALUATION METHODSI ntroduction167 Comparing and ContrastingDifferent Approaches to QUALITATIVE Evaluation168 Understanding the Issue of Paradigms169 The Pragmatic Approach172 QUALITATIVE EVALUATION METHODS : Some Basics172 Key Differences Between Qualitativeand Quantitative EVALUATION Approaches174 Structuring QUALITATIVE Program Evaluations179 Identifying EVALUATION Questions and Issues in Advance179 Identifying Research Designs and Appropriate Comparisons180 Identifying Appropriate Samples181 Structuring Data Collection Instruments184 Collecting and Coding QUALITATIVE Data187 The Credibility andGeneralizability of QUALITATIVE 5/26/2005 7:00 PM Page 165 Connecting QUALITATIVE EvaluationMethods to Performance Measurement194 The Power of Case Studies196 Summary197 Discussion Questions198 References198166 PROGRAM EVALUATION AND PERFORMANCE 5/26/2005 7:00 PM Page 166 INTRODUCTION This chapter introduces a different approach to program EVALUATION onethat has emerged in parallel to the more structured, quantitative approachthat has been elaborated in Chapters 2, 3, and 4.

2 This chapter will show howqualitative EVALUATION methodscan be incorporated into the range ofoptions available to evaluators and their clients, and will offer some com-parisons between the two approaches. In general, QUALITATIVE approaches,such as interviews and focus groupdiscussions, are more open-ended thanquantitative METHODS , and are most valuable in collecting and analyzing datathat do not readily reduce into numbers. QUALITATIVE METHODS are particularlyuseful for exploratory research and participatory, or empowerment, research (see Fetterman, 1996) involves significant collabora-tion between the evaluator and stakeholders during most or all of the stepsin the EVALUATION process, from the planning and design to the final inter-pretation and recommendations.

3 Chapter 11 includes further information onempowerment EVALUATION in its discussion of the relationship between pro-gram EVALUATION and performance is worth reiterating that the key issues in deciding on which methodor METHODS to use for any EVALUATION are the context of the situation and theevaluation questions that need to be addressed. QUALITATIVE METHODS can beused in various stages of an EVALUATION : Determining the focus of the EVALUATION Evaluating the implementation or the process of a program Determining improvements and changes to a programTo introduce QUALITATIVE EVALUATION METHODS , it is important to first elab-orate on the diversity of approaches even within the theory and practice ofqualitative EVALUATION .

4 QUALITATIVE EVALUATION approaches differ from eachother on at least two important fronts: their philosophical beliefs about howand what we can know about the kinds of situations evaluators typically face(these are called epistemological beliefs); and their methodologies, thatis, the ways that evaluations are organized and conducted. In this chapter, wewill learn about some of the key philosophical differences among qualitativeevaluation approaches, but will spend more time focusing on the ways thatqualitative METHODS can be QUALITATIVE EVALUATION METHODS 5/26/2005 7:00 PM Page 167 COMPARING AND CONTRASTING DIFFERENTAPPROACHES TO QUALITATIVE EVALUATION168 PROGRAM EVALUATION AND PERFORMANCE MEASUREMENTWhen QUALITATIVE EVALUATION approaches emerged as alternatives to the then-dominant social scientific approachto EVALUATION in the 1970s, proponentsof these new ways of evaluating programs were a part of a much broadermovement to re-make the foundations and the practice of social research has a long history, particularly in disciplines like anthro-pology and sociology, and there have been important changes over time in theways that QUALITATIVE researchers see their enterprise.

5 There is more diversitywithin QUALITATIVE EVALUATION approaches than within quantitative approaches:A significant difference between QUALITATIVE and quantitative METHODS isthat, while the latter have established a working philosophical consen-sus, the former have not. This means that quantitative researchers cantreat methodology as a technical matter. The best solution is one whichmost effectively and efficiently solves a given problem. The same is nottrue for QUALITATIVE research where proposed solutions to methodologi-cal problems are inextricably linked to philosophical assumptions andwhat counts as an appropriate solution from one position is fatallyflawed from another.

6 (Murphy, Dingwall, Greatbatch, et al., 1998, p. 58)Denzin and Lincoln (2000) summarize the history of QUALITATIVE researchin their introduction to the Handbook of QUALITATIVE offer aninterpretation of the history of QUALITATIVE research in North America as com-prising seven moments, beginning with traditional anthropological research(1800s to about 1950), characterized by lone anthropologists spending timein other cultures and then rendering their findings in objective accounts ofthe values, beliefs and behaviors of the natives. In their definition of qualita-tive research Denzin and Lincoln (2000) include the following: QUALITATIVE research is a situated activity that locates the observer in researchers study things in their natural settings,attempting to make sense of, or to interpret, phenomena in terms of themeanings people bring to them.

7 (p. 3)and later continue: QUALITATIVE researchers stress the socially constructed nature of reality,the intimate relationship between the researcher and what is studied, 5/26/2005 7:00 PM Page 168and the situational constraints that shape inquiry. Such researchersemphasize the value-laden nature of enquiry. They seek answers toquestions that stress howsocial experience is created and given mean-ing. In contrast, quantitative studies emphasize the measurement andanalysis of causal relationships between variables, not processes. (p. 8)Understanding the Issue of ParadigmsIn the field of program EVALUATION , academics and practitioners in the1970s were increasingly under pressure to justify the then-dominant socialscience-based model as a way of thinking about and conducting about the relevance and usefulness of highly structured evalua-tions (often experiments) were being raised by clients and by academicsalike.

8 An alternative paradigm was emerging, based on different assumptions,different ways of gathering information, different ways of interpreting thatinformation, and finally, different ways of reporting EVALUATION findings the whole, the QUALITATIVE research approach embraces a different viewof research than the positivist rational approach traditionally taken by mostquantitative researchers. Thomas S. Kuhn (1962), in his revolutionary bookThe Structure of Scientific Revolutions,asserted that when scientists dis-cover a new way of looking at phenomena, they literally see the world in adifferent developed and popularized the notion of a paradigm, aself-contained perceptual and theoretical structure akin to a belief Kuhn was describing the change in world view that happened inphysics when Einstein s relativity theory began its ascendancy at the turn of the20th century, he used language and examples that invited generalizing to otherfields.

9 In fact, because his book was written in a nontechnical way, it became amajor contribution to the widespread and continuing process of questioningthe foundations of our knowledge in the social sciences and , for Kuhn, were at least partly is,adherence to one paradigm, and its attendant way of seeing the world, wouldnot be translatable into a different paradigm. Proponents of dissimilar para-digms would experience an inability to communicate with their counter-parts. They would talk past each other, because they would use differentwords and literally see different things even when they were pointing to thesame Russell Hanson (1958) illustrated this problem with a series ofoptical illusions and puzzles.

10 A well-known one is reproduced here to makeseveral points about seeing. Figure is a line drawing of a person. LookApplying QUALITATIVE EVALUATION METHODS 5/26/2005 7:00 PM Page 169carefully at the drawing and see if you can discern the old woman s face. Sheis facing the left side of the page, her nose is prominent, her lips are a line,and she is wearing a hood that exposes the hair on her , look at the same drawing again and see if you can discern theyoung woman s face. She is facing away from you, and is showing you the leftside of her face. You cannot see her nose, but you can see her left ear. Herjaw and her hair are visible as is her necklace. A hood that loosely covers herhead exposes the left side of her , people find it easier to see the old woman.