QUANTITATIVE AND QUALITATIVE METHODS - INTRAC

1 INTRAC 2020 QUANTITATIVE AND QUALITATIVE METHODS QUANTITATIVE and QUALITATIVE METHODS generate different types of data. QUANTITATIVE data is expressed as numbers; QUALITATIVE data is expressed as words. QUANTITATIVE and QUALITATIVE METHODS can be combined in many ways to build on the strengths of both, and minimise their relative weaknesses. There is a growing consensus that both are important. This has led to an increased interest in mixed METHODS evaluations. QUANTITATIVE and QUALITATIVE METHODS generate different types of data. In general, QUANTITATIVE METHODS result in QUANTITATIVE data, whilst QUALITATIVE METHODS produce QUALITATIVE data. QUANTITATIVE data is expressed in numbers ( units, prices, proportions, rates of change and ratios). QUALITATIVE data is expressed as words ( statements, paragraphs, stories, case studies and quotations).

2 As well as resulting in different forms of data, QUANTITATIVE and QUALITATIVE METHODS pursue different approaches to collect, analyse and use data. Many of these are covered elsewhere in the M&E Universe, under separate papers. The table opposite summarises some of the broad differences. Most monitoring and evaluation (M&E) systems generate a mixture of QUANTITATIVE and QUALITATIVE information. For example, almost all CSOs produce budgetary reports ( QUANTITATIVE information) and regular narrative progress updates ( QUALITATIVE information). However, within the evaluation and research communities there have often been fierce debates around whether QUANTITATIVE or QUALITATIVE METHODS are most appropriate for assessing change, and under which conditions. People engaged in these debates can be divided into three groups. The first value QUANTITATIVE information most, whilst recognising that QUALITATIVE information can be an important and useful supplement.

3 The second group are more interested in QUALITATIVE information, whilst accepting that numbers also have a role to play. The third, and now by far the largest, group see QUANTITATIVE and QUALITATIVE METHODS as complementary. This group recognise that both have their own particular strengths and weaknesses, and that they can be more effective when used in combination. Strengths of QUANTITATIVE METHODS Some of the strengths of QUANTITATIVE METHODS , compared to QUALITATIVE METHODS , are given below. However, these are generalisations, and there are often exceptions. The differences are summarised in the diagram on the next page. QUANTITATIVE METHODS can easily cope with very large numbers of cases. This is because it is generally much easier to process numeric data. For example, it is much simpler to work out the average number of days lost due to illness across a very large number of people QUANTITATIVE QUALITATIVE Data Numbers Words Typical METHODS of data collection Predefined options and closed questions in surveys, direct measurement, digital data collection Open-ended questions in surveys and interviews, focus group discussions, observation, case studies Analysis Statistical data METHODS (averages, correlations, regression analysis) Summarisation, reduction and scoring.

4 In -depth analysis of individual cases Sampling Large, random samples Purposive (deliberate) sampling of most interesting cases Indicators Specific, measurable, numeric indicators Broadly defined QUALITATIVE indicators or questions Milestones and targets Easy to define and to communicate Hard to define and communicate Baselines Numeric collection and presentation of data Narrative collection and presentation of data Control or comparison groups Often used in experimental or quasi-experimental METHODS Rarely used in QUALITATIVE inquiry Typical monitoring questions How much? How many? How often? How or why did something happen? For whom? Data storage and processing Data stored as numbers; large amount of automatic processing Data stored as words or as attached reports; less automatic processing INTRAC 2020 than to summarise their perceptions of how illnesses are transmitted.

5 This is largely due to the difficulties of handling and processing large amounts of QUALITATIVE data. Partly because they are better able to handle large numbers of cases, QUANTITATIVE METHODS are better able to provide a broad overview of a situation, and to make generalisations across populations. For example, if an evaluation collects information on the education level of heads of households, and how many of their children go to school, it is possible to make general statements, such as people who are better educated are more (or less) likely to send their children to school. QUALITATIVE METHODS are less able to do this as they are often more focused on taking an in-depth look at individual cases. There are standard ways of analysing QUANTITATIVE data, such as calculating averages, producing correlations, or using regression analysis.

6 These work according to agreed and established rules, which can be taught and replicated. Providing the methodology of a QUANTITATIVE study or evaluation is transparent, the accuracy and reliability of results can be measured. This is not so in QUALITATIVE inquiry, where the rules are not so well established. In theory, anyone analysing the same QUANTITATIVE data should be able to come up with the same findings. QUANTITATIVE METHODS should work irrespective of who collects and analyses the data, and data should be comparable across multiple locations, even if many different people are involved in data collection. This is not true in QUALITATIVE inquiry, where analysis is much more dependent on the skills, honesty and integrity of those collecting and analysing the information (Patton 1990). It is much easier to aggregate and summarise numeric data than large amounts of QUALITATIVE data.

7 For example, a computer can calculate the average weightand height of 1,000 babies as easily and as quickly as it can 10 babies. By contrast, QUALITATIVE data cannot be aggregated, and as the number of cases increases it becomes much harder to summarise. Summaries of QUALITATIVE data also have the potential to be more subjective, and open to bias. It is much easier to record, store and process QUANTITATIVE data than QUALITATIVE data. Numbers are easy to enter onto a database, and can be processed automatically. It takes longer to record and store QUALITATIVE information, and it is usually harder to process it automatically. Because of this, fewer CSOs have systems for storing and analysing QUALITATIVE information (see ITAD 2014). Many (not all) decision-makers, especially government representatives, trust QUANTITATIVE analysis more than they do QUALITATIVE analysis.

8 This is largely because they feel that: a) it provides a better overview of a situation and allows for generalisations; b) findings are less dependent on the biases of individual evaluators or researchers; and c) the rules are standardised and accepted. Some decision-makers are deeply suspicious of QUALITATIVE METHODS , and feel they can easily be manipulated to suit the purposes of different evaluators, researchers or organisations. Strengths of QUALITATIVE METHODS QUALITATIVE METHODS also have many strengths. These are often mirror images of QUANTITATIVE METHODS . Where QUANTITATIVE METHODS are seen as being strong, QUALITATIVE METHODS are seen as (relatively) weaker. And where QUANTITATIVE METHODS are seen as weaker, QUALITATIVE METHODS are seen as (relatively) stronger. Some of the main strengths of QUALITATIVE METHODS are listed below, and are summarised in the diagram above.

9 As with the section on QUANTITATIVE METHODS , these are general rules only, and there are often exceptions. QUANTITATIVE METHODS .. QUALITATIVE METHODS .. Can handle a very large number of cases Cannot easily deal with a large number of cases Generate findings that may be treated with suspicion Generate findings valued by many decision-makers Provide a broad overview of a situation Do not always provide an overview Produce generalisable findings Produce findings that are more specific to context Use established, standardised METHODS of analysis Do not have such well-established rules for analysis Enable comparison across different data sources Generate data that is harder to compare Enable aggregation and summarisation Do not allow for aggregation or easy summarisation Result in data that is easy to record, store and process Generate data that is hard to store and process INTRAC 2020 QUALITATIVE METHODS can be used no matter how few cases there are.

10 And can work well with a small- or medium-sized number of cases (anywhere between 1 and 100). This is because they are concerned with examining a selection of cases in-depth. This is why CSOs generally use QUALITATIVE METHODS when evaluating work such as advocacy or capacity development, where there may only be a few organisations or policies to consider. By contrast, in order to work properly, many statistical METHODS require a sufficient number of cases. In some situations, QUANTITATIVE analysis can show what has changed, but is unable to explain why or how it has changed. On the other hand, QUALITATIVE analysis METHODS often shed light on the processes that led to change. This means they are usually considered better if the need is to find out why something happened, or how it happened. QUALITATIVE METHODS can almost always be used to investigate causality (assesing how far a change was caused by an organisation or programme) whereas QUANTITATIVE METHODS can only do so in a narrow range of circumstances.