Approaches to the Analysis of Survey Data

1 Approaches to the Analysis of Survey data March 2001 The University of Reading Statistical Services Centre Biometrics Advisory and Support Service to DFID 2001 Statistical Services Centre, The University of Reading, UK SSC 2001 Approaches to the Analysis of Survey data 3 Contents 1. Preparing for the Analysis 5 Introduction 5 data Types 6 data Structure 7 Stages of Analysis 9 Population Description as the Major Objective 11 Comparison as the Major Objective 12 When Weighting Matters 13 Coding 14 Ranking & Scoring 15 2. Doing the Analysis 17 Approaches 17 One-Way Tables 18 Cross-Tabulation: Two-Way & Higher-Way Tables 18 Tabulation & the Assessment of Accuracy 19 Multiple Response data 20 Profiles 21 Looking for Respondent Groups 22 Indicators 23 Validity 25 Summary 26 Next Steps 26 4 SSC 2001 Approaches to the Analysis of Survey data SSC 2001 Approaches to the Analysis of Survey data 5 1.

2 Preparing for the Analysis Introduction This guide is concerned with some fundamental ideas of Analysis of data from surveys. The discussion is at a statistically simple level; other more sophisticated statistical Approaches are outlined in our guide Modern Methods of Analysis . Our aim here is to clarify the ideas that successful data analysts usually need to consider to complete a Survey Analysis task purposefully. An ill-thought-out Analysis process can produce incompatible outputs and many results that never get discussed or used. It can overlook key findings and fail to pull out the subsets of the sample where clear findings are evident. Our brief discussion is intended to assist the research team in working systematically; it is no substitute for clear-sighted and thorough work by researchers.

3 We do not aim to show a totally na ve analyst exactly how to tackle a particular set of Survey data . However, we believe that where readers can undertake basic Survey Analysis , our recommendations will help and encourage them to do so better. Chapter 1 outlines a series of themes, after an introductory example. Different data types are distinguished in section Section looks at data structures; simple if there is one type of sampling unit involved, and hierarchical with communities, households and individuals. In section we separate out three stages of Survey data handling exploration, Analysis and archiving which help to define expectations and procedures for different parts of the overall process.

4 We contrast the research objectives of description or estimation (section ), and of comparison (section ) and what these imply for Analysis . Section considers when results should be weighted to represent the population depending on the extent to which a numerical value is or is not central to the interpretation of Survey results. In section we outline the coding of non-numerical responses. The use of ranked data is discussed in brief in section In Chapter 2 we look at the ways in which researchers usually analyse Survey data . We focus primarily on tabular methods, for reasons explained in section Simple one-way tables are often useful as explained in section Cross-tabulations (section ) can take many forms and we need to think which are appropriate.

5 Section discusses issues about accuracy in relation to two- and multi-way tables. In section we briefly discuss what to do when several responses can be selected in response to one question. 6 SSC 2001 Approaches to the Analysis of Survey data Cross-tabulations can look at many respondents, but only at a small number of questions, and we discuss profiling in section , cluster Analysis in section , and indicators in sections and data Types Introductory Example: On a nominal scale the categories recorded, usually counted, are described verbally. The scale has no numerical characteristics. If a single one-way table resulting from simple summarisation of nominal (also called categorical) scale data contains frequencies:- Christian Hindu Muslim Sikh Other 29 243 117 86 25 there is little that can be done to present exactly the same information in other forms.

6 We could report highest frequency first as opposed to alphabetic order, or reduce the information in some way if one distinction is of key importance compared to the others:- Hindu Non-Hindu 243 257 On the other hand, where there are ordered categories, the sequence makes sense only in one, or in exactly the opposite, order:- Excellent Good Moderate Poor Very Bad 29 243 117 86 25 We could reduce the information by combining categories as above, but also we can summarise, somewhat numerically, in various ways. For example, accepting a degree of arbitrariness, we might give scores to the categories:- Excellent Good Moderate Poor Very Bad 5 4 3 2 1 and then produce an average score a numerical indicator for the sample of:- 29 5 + 243 4 + 117 3 + 86 2 + 25 1 = 29 + 243 + 117 + 86 + 25 This is an analogue of the arithmetical calculation we would do if the categories really were numbers family sizes.

7 SSC 2001 Approaches to the Analysis of Survey data 7 The same average score of could arise from differently patterned data from rather more extreme results:- Excellent Good Moderate Poor Very Bad 79 193 117 36 75 Hence, as with any other indicator, this average only represents one feature of the data and several summaries will sometimes be needed. A major distinction in statistical methods is between quantitative data and the other categories exemplified above. With quantitative data , the difference between the values from two respondents has a clearly defined and incontrovertible meaning It is 5C hotter now than it was at dawn or You have two more children than your sister.

8 Commonplace statistical methods provide many well-known Approaches to such data , and are taught in most courses, so we give them only passing attention here. In this guide we focus primarily on the other types of data , coded in number form but with less clear-cut numerical meaning, as follows. Binary yes/no data can be coded in 1/0 form; while purely categorical or nominal data caste or ethnicity may be coded 1, 2, using numbers that are just arbitrary labels and cannot be added or subtracted. It is also common to have ordered categorical data , where items may be rated Excellent, Good, Poor, Useless, or responses to attitude statements may be Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree. With ordered categorical data the number labels should form a rational sequence, because they have some numerical meaning scores of 4, 3, 2, 1 for Excellent through to Useless.

9 Such data supports limited quantitative Analysis , and is often referred to by statisticians as qualitative this usage does not imply that the elicitation procedure must satisfy a purist s restrictive perception of what constitutes qualitative research methodology. data Structure SIMPLE Survey data STRUCTURE: the data from a single-round Survey , analysed with limited reference to other information, can often be thought of as a flat rectangular file of numbers, whether the numbers are counts/measurements, or codes, or a mixture. In a structured Survey with numbered questions, the flat file has a column for each question, and a row for each respondent, a convention common to almost all standard statistical packages.

10 If the data form a perfect rectangular grid with a number in every cell, Analysis is made relatively easy, but there are many reasons why this will not always be the case and flat file data will be incomplete or irregular. Most importantly:- 8 SSC 2001 Approaches to the Analysis of Survey data Surveys often involve skip questions where sections are missed out if irrelevant details of spouse s employment do not exist for the unmarried. These arise legitimately, but imply different subsets of people respond to different questions. Contingent questions , where not everyone qualifies to answer, often lead to inconsistent-seeming results for this reason. If the overall sample size is just adequate, the subset who qualify for a particular set of contingent questions may be too small to analyse in the detail required.