Qualitative Comparative Analysis – A Rigorous Qualitative …

1 Qualitative Comparative Analysis . A Rigorous Qualitative Method for Assessing Impact Coffey How To June 2015. Summary QCA is a case based method which allows evaluators to identify different combinations of factors that are critical to a given outcome, in a given context. This allows for a more nuanced understanding of how different combinations of factors can lead to success, and the influence context can have on success. QCA allows evaluators to test theories of change and answer the question what works best, why and under what circumstances' in a way that emerges directly from the empirical Analysis , that can be replicated by other researchers, and is generalizable to other contexts.

2 While it isn't appropriate for use in all circumstances and has limitations, QCA also has certain unique strengths including qualitatively assessing impact and identifying multiple pathways to achieving change which make it a valuable addition to the evaluation toolkit. Authors Carrie Baptist and Barbara Befani What is Qualitative Comparative Analysis (QCA) and what can it do? As evaluators, we are often asked not only to assess Qualitative Comparative Analysis (QCA) is a whether a project or intervention has worked, but why, case-based method that enables evaluators to how and under what circumstances.

3 This contextual understanding is critical for understanding not only whether systematically compare cases, identifying key a project has been successful in a particular context, but factors which are responsible for the success of an also whether and under what conditions it can be scaled up intervention. or replicated. As a Comparative method, QCA doesn't work with a The use of case-based approaches, like case studies, is a well-established method for evaluators and researchers single case it needs to compare factors at work across who wish to draw out narratives and investigate a theme a number of cases in order to tease out which factors in-depth, probing not only what has occurred but why, how are most important for a given outcome.

4 However, when and in what context. However, these types of approaches done correctly the findings generated using QCA are also have weaknesses; while they provide great depth in generalizable: insights from one context or project can be a particular outcome in a particular context, it's unclear transferred to another. QCA is also a Rigorous method in that how generalizable that learning is to other projects or when the Analysis process is made transparent it can be contexts. In addition, if not carried out systematically and replicated by anyone and produce the same findings.

5 Transparently, findings from case studies are sometimes considered a weaker' type of evidence, as they are not To put it a bit more technically, QCA is potentially strong replicable by external researchers (whereas quantitative on both external and internal validity while still being Analysis generates results which in many cases can be a Qualitative method: it works with social concepts, consistently replicated externally). constructs and narratives. In this way, QCA seeks to combine the strengths This contextual, complex' approach seems fitting for of Qualitative and quantitative research, by development interventions, which usually work differently in different contexts and geographical areas, and for which linking theory and evidence while also providing there is unlikely to be a single silver bullet' which guarantees increased measurement precision1.

6 Success. Additionally, while statistical methods may be more In this sense, QCA usefully refocuses attention appropriate when large datasets are available, QCA does not need a high number of cases in order for the findings to back onto context, systems and institutions, and be significant 2. away from essentialised or contextless best practices'. Put simply, QCA allows evaluators to answer the question Testing Theories of Change and engaging what works best, why and under what circumstances . with complexity quite literally, in a way that emerges directly from the empirical Analysis ; that can be replicated by other QCA provides an innovative way to test project or researchers, and is generalizable to other contexts.

7 Programme theories of change. A detailed theory of change will identify not only inputs, outputs, outcomes and impact, but also key assumptions and external dynamics How has it been used to-date? which may impact success. QCA allows evaluators to test the relationships between key assumptions and external QCA was first developed by an American social scientist, factors identified in a theory of change, and a given Charles Ragin, in 19875. To date, it has been used outcome or impact across a set of cases. QCA will identify most extensively in the academic sphere, primarily in which, of the factors tested, are necessary or sufficient to Comparative sociology and politics, but more recently it obtain a successful outcome or impact.

8 Has been applied to policy analysis6 and evaluation. After a few isolated attempts (see for example Befani et al. 2007), Generally, a single factor isn't necessary or sufficient for evaluators have started using QCA more often over the past success rather a combination of factors (also known as 2 -3 years, including in meta evaluations of approaches a package') is needed. QCA can identify the package(s) of to evaluating interventions targeting violence against factors necessary or sufficient for a successful outcome, women and girls7, evaluations of regional governance, and which factors within those combinations are most empowerment and accountability programming8 and critical in which contexts3.

9 Evaluations of public health interventions9. In other words, QCA can identify different combinations However, the use of QCA in evaluations is still new, and the of factors that are critical to a given outcome, in a given advantages and pitfalls of this method for evaluators are context4. This allows for a more nuanced understanding of still being established. While the potential of QCA is starting how different combinations of factors can lead to success, to become increasingly clear in the evaluation community, and the influence context can have on success.

10 There is still no comprehensive and well-established guide on how best to use QCA for evaluation, or the strategies It also allows evaluators to move away from a evaluators can employ to meet the challenges posed by linear understanding of how change happens QCA. While it is beyond the scope of this brief to provide and incorporate multiple paths and combinations of ingredients', which are different but equally 5. Ragin, Charles C. The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. University of California Press 1987. relevant for a successful outcome.