APPLYING QUALITATIVE EVALUATION METHODS

1 165 CHAPTER5 APPLYING QUALITATIVEEVALUATION METHODSI ntroduction167 Comparing and ContrastingDifferent Approaches to QUALITATIVE Evaluation168 Understanding the Issue of Paradigms169 The Pragmatic Approach172 QUALITATIVE EVALUATION METHODS : Some Basics172 Key Differences Between Qualitativeand Quantitative EVALUATION Approaches174 Structuring QUALITATIVE Program Evaluations179 Identifying EVALUATION Questions and Issues in Advance179 Identifying Research Designs and Appropriate Comparisons180 Identifying Appropriate Samples181 Structuring Data Collection Instruments184 Collecting and Coding QUALITATIVE Data187 The Credibility andGeneralizability of QUALITATIVE 5/26/2005 7:00 PM Page 165 Connecting QUALITATIVE EvaluationMethods to Performance Measurement194 The Power of Case Studies196 Summary197 Discussion Questions198 References198166 PROGRAM EVALUATION AND PERFORMANCE 5/26/2005 7:00 PM Page 166 INTRODUCTION This chapter introduces a different approach to program EVALUATION onethat has emerged in parallel to the more structured, quantitative approachthat has been elaborated in Chapters 2, 3, and 4.

2 This chapter will show howqualitative EVALUATION methodscan be incorporated into the range ofoptions available to evaluators and their clients, and will offer some com-parisons between the two approaches. In general, QUALITATIVE approaches,such as interviews and focus groupdiscussions, are more open-ended thanquantitative METHODS , and are most valuable in collecting and analyzing datathat do not readily reduce into numbers. QUALITATIVE METHODS are particularlyuseful for exploratory research and participatory, or empowerment, research (see Fetterman, 1996) involves significant collabora-tion between the evaluator and stakeholders during most or all of the stepsin the EVALUATION process, from the planning and design to the final inter-pretation and recommendations. Chapter 11 includes further information onempowerment EVALUATION in its discussion of the relationship between pro-gram EVALUATION and performance is worth reiterating that the key issues in deciding on which methodor METHODS to use for any EVALUATION are the context of the situation and theevaluation questions that need to be addressed.

3 QUALITATIVE METHODS can beused in various stages of an EVALUATION : Determining the focus of the EVALUATION Evaluating the implementation or the process of a program Determining improvements and changes to a programTo introduce QUALITATIVE EVALUATION METHODS , it is important to first elab-orate on the diversity of approaches even within the theory and practice ofqualitative EVALUATION . QUALITATIVE EVALUATION approaches differ from eachother on at least two important fronts: their philosophical beliefs about howand what we can know about the kinds of situations evaluators typically face(these are called epistemological beliefs); and their methodologies, thatis, the ways that evaluations are organized and conducted. In this chapter, wewill learn about some of the key philosophical differences among qualitativeevaluation approaches, but will spend more time focusing on the ways thatqualitative METHODS can be QUALITATIVE EVALUATION METHODS 5/26/2005 7:00 PM Page 167 COMPARING AND CONTRASTING DIFFERENTAPPROACHES TO QUALITATIVE EVALUATION168 PROGRAM EVALUATION AND PERFORMANCE MEASUREMENTWhen QUALITATIVE EVALUATION approaches emerged as alternatives to the then-dominant social scientific approachto EVALUATION in the 1970s, proponentsof these new ways of evaluating programs were a part of a much broadermovement to re-make the foundations and the practice of social research has a long history, particularly in disciplines like anthro-pology and sociology, and there have been important changes over time in theways that QUALITATIVE researchers see their enterprise.

4 There is more diversitywithin QUALITATIVE EVALUATION approaches than within quantitative approaches:A significant difference between QUALITATIVE and quantitative METHODS isthat, while the latter have established a working philosophical consen-sus, the former have not. This means that quantitative researchers cantreat methodology as a technical matter. The best solution is one whichmost effectively and efficiently solves a given problem. The same is nottrue for QUALITATIVE research where proposed solutions to methodologi-cal problems are inextricably linked to philosophical assumptions andwhat counts as an appropriate solution from one position is fatallyflawed from another. (Murphy, Dingwall, Greatbatch, et al., 1998, p. 58)Denzin and Lincoln (2000) summarize the history of QUALITATIVE researchin their introduction to the Handbook of QUALITATIVE offer aninterpretation of the history of QUALITATIVE research in North America as com-prising seven moments, beginning with traditional anthropological research(1800s to about 1950), characterized by lone anthropologists spending timein other cultures and then rendering their findings in objective accounts ofthe values, beliefs and behaviors of the natives.

5 In their definition of qualita-tive research Denzin and Lincoln (2000) include the following: QUALITATIVE research is a situated activity that locates the observer in researchers study things in their natural settings,attempting to make sense of, or to interpret, phenomena in terms of themeanings people bring to them. (p. 3)and later continue: QUALITATIVE researchers stress the socially constructed nature of reality,the intimate relationship between the researcher and what is studied, 5/26/2005 7:00 PM Page 168and the situational constraints that shape inquiry. Such researchersemphasize the value-laden nature of enquiry. They seek answers toquestions that stress howsocial experience is created and given mean-ing. In contrast, quantitative studies emphasize the measurement andanalysis of causal relationships between variables, not processes. (p. 8)Understanding the Issue of ParadigmsIn the field of program EVALUATION , academics and practitioners in the1970s were increasingly under pressure to justify the then-dominant socialscience-based model as a way of thinking about and conducting about the relevance and usefulness of highly structured evalua-tions (often experiments) were being raised by clients and by academicsalike.

6 An alternative paradigm was emerging, based on different assumptions,different ways of gathering information, different ways of interpreting thatinformation, and finally, different ways of reporting EVALUATION findings the whole, the QUALITATIVE research approach embraces a different viewof research than the positivist rational approach traditionally taken by mostquantitative researchers. Thomas S. Kuhn (1962), in his revolutionary bookThe Structure of Scientific Revolutions,asserted that when scientists dis-cover a new way of looking at phenomena, they literally see the world in adifferent developed and popularized the notion of a paradigm, aself-contained perceptual and theoretical structure akin to a belief Kuhn was describing the change in world view that happened inphysics when Einstein s relativity theory began its ascendancy at the turn of the20th century, he used language and examples that invited generalizing to otherfields.

7 In fact, because his book was written in a nontechnical way, it became amajor contribution to the widespread and continuing process of questioningthe foundations of our knowledge in the social sciences and , for Kuhn, were at least partly is,adherence to one paradigm, and its attendant way of seeing the world, wouldnot be translatable into a different paradigm. Proponents of dissimilar para-digms would experience an inability to communicate with their counter-parts. They would talk past each other, because they would use differentwords and literally see different things even when they were pointing to thesame Russell Hanson (1958) illustrated this problem with a series ofoptical illusions and puzzles. A well-known one is reproduced here to makeseveral points about seeing. Figure is a line drawing of a person. LookApplying QUALITATIVE EVALUATION METHODS 5/26/2005 7:00 PM Page 169carefully at the drawing and see if you can discern the old woman s face.

8 Sheis facing the left side of the page, her nose is prominent, her lips are a line,and she is wearing a hood that exposes the hair on her , look at the same drawing again and see if you can discern theyoung woman s face. She is facing away from you, and is showing you the leftside of her face. You cannot see her nose, but you can see her left ear. Herjaw and her hair are visible as is her necklace. A hood that loosely covers herhead exposes the left side of her , people find it easier to see the old woman. But once you see the young woman, she will be there every time you look for point of Figure is to show you that the same information can beinterpreted in different ways. Further, the patterns of information are incom-mensurable with each other. When you are seeing the old woman, youare not (at that moment) seeing the young woman. Kuhn was arguing thatscientists, too, can see the same information and interpret it the QUALITATIVE METHODS arena, one key paradigm, often called theconstructivistparadigm, has been developed and articulated by a numberof central contributors.

9 Egon Guba and Yvonna Lincoln (2001) are twowell-known scholars who have made major contributions to constructivistresearch approaches. In explaining relativism, one of the fundamentalassumptions of the constructivist paradigm, they state that human (semiotic)170 PROGRAM EVALUATION AND PERFORMANCE MEASUREMENTF igure Old Woman or the Young Woman? 5/26/2005 7:00 PM Page 170sense-making that organizes experience so as to render it into apparentlycomprehensible, understandable, and explainable form, is an act of construaland is independent of any foundational relativism there canbe no objective truth (p. 1, original underlining).Guba and Lincoln are among those who have stated a belief that the sci-entific approach to program EVALUATION and the constructivist approach arefundamentally different, and at least partially incommensurable, paradigms:It is not appropriate to mix and match paradigms in conducting anevaluation, for example, utilizing both scientific (positivist) and con-structivist propositions within the same study.

10 This is not a call for purity nor is it intended to be exclusionary. It is simply a caveat thatmixing paradigms may well result in nonsense approaches and conclu-sions. (Guba & Lincoln, 2001, p. 2)However, other influential evaluators, including Michael Patton, whileacknowledging the continued existence of paradigmatic differences betweensocial scientific evaluators and some QUALITATIVE research approaches, haveargued that the philosophical differences have been at least substantiallysubmerged by more practical concerns:The trends and factors just reviewed suggest that the paradigms debatehas withered substantially. The focus has shifted to methodologicalappropriateness rather than orthodoxy, methodological creativity ratherthan rigid adherence to a paradigm, and methodological flexibilityrather than conformity to a narrow set of rules. (Patton, 1997, p.)