Fundamentals of Survey Research Methodology

1 MP 05W0000077 MITRE PRODUCT Fundamentals of Survey Research Methodology April 2005 Priscilla A. Glasow (25988) Division: Department: W800 W804 . Washington C3 Center McLean, Virginia MITRE Department Approval: Edward F. Gonzalez Table of Contents Section Page1. What is Survey Research ?

2 1-1 Survey 1-1 Survey 1-2 Definition of Terms .. 1-2 2. The Survey Process 2-1 Survey 2-1 Sample 2-1 Sample 2-2 Degree of 2-2 Statistical 2-2 Effect Size as a Determinant of 2-3 Sample Size as a Determinant of 2-3 Access to 2-3 Stratification and 2-3 Units of 2-3 Choice of Survey 2-4 Written 2-4 Verbal 2-4 Mixed Mode 2-5 Survey Instrument 2-5 Standards for Good Survey 2-5 Question 2-5 Feasible and 2-6 Additional 2-6 Biased 2-6

3 Biased 2-6 General 2-6 Cognitive Implications of Biased 2-7 Facets of Biased 2-7 Types of Survey 2-7 Open-ended 2-7 Closed-ended 2-7 Questions that Describe and Evaluate People, Places and 2-8 Questions that Measure Responses to Ideas, Analyses or 2-9 Questions that Measure 2-9 iii Section Subjective Responses to Survey 2-9 Cognitive Tasks Required for Survey 2-9 Sources of Measurement Survey Data Analysis and Reporting Survey 3.

4 Conclusion 3-1 List of References RE-1 Appendix . A Checklist for Survey Assessment A-1 Survey A-1 Population and A-1 Survey Instrument .. A-1 Study A-2 Data Analysis A-2 Distribution List DI-1 iv Section 1 What is Survey Research ? Survey Research is used: to answer questions that have been raised, to solve problems that have been posed or observed, to assess needs and set goals, to determine whether or not specific objectives have been met, to establish baselines against which future comparisons can be made, to analyze trends across time, and generally, to describe what exists, in what amount, and in what context.

5 (Isaac & Michael, 1997, p. 136) Kraemer (1991) identified three distinguishing characteristics of Survey Research (p. xiii). First, Survey Research is used to quantitatively describe specific aspects of a given population. These aspects often involve examining the relationships among variables. Second, the data required for Survey Research are collected from people and are, therefore, subjective. Finally, Survey Research uses a selected portion of the population from which the findings can later be generalized back to the population.

6 In Survey Research , independent and dependent variables are used to define the scope of study, but cannot be explicitly controlled by the researcher. Before conducting the Survey , the researcher must predicate a model that identifies the expected relationships among these variables. The Survey is then constructed to test this model against observations of the phenomena. In contrast to Survey Research , a Survey is simply a data collection tool for carrying out Survey Research . Pinsonneault and Kraemer (1993) defined a Survey as a means for gathering information about the characteristics, actions, or opinions of a large group of people (p.)

7 77). Surveys can also be used to assess needs, evaluate demand, and examine impact (Salant & Dillman, 1994, p. 2). The term Survey instrument is often used to distinguish the Survey tool from the Survey Research that it is designed to support. Survey Strengths Surveys are capable of obtaining information from large samples of the population. They are also well suited to gathering demographic data that describe the composition of the sample (McIntyre, 1999, p. 74). Surveys are inclusive in the types and number of variables that can be studied, require minimal investment to develop and administer, and are relatively easy for making generalizations (Bell, 1996, p.

8 68). Surveys can also elicit information about attitudes that are otherwise difficult to measure using observational techniques (McIntyre, 1999, p. 75). It is important to note, however, that surveys only provide estimates for the true population, not exact measurements (Salant & Dillman, 1994, p. 13). 1-1 Survey Weaknesses Pinsonneault and Kraemer (1993) noted that surveys are generally unsuitable where an understanding of the historical context of phenomena is required. Bell (1996) observed that biases may occur, either in the lack of response from intended participants or in the nature and accuracy of the responses that are received.

9 Other sources of error include intentional misreporting of behaviors by respondents to confound the Survey results or to hide inappropriate behavior. Finally, respondents may have difficulty assessing their own behavior or have poor recall of the circumstances surrounding their behavior. Definition of Terms Verbal surveys are often known as interviews and written surveys are questionnaires. Reliability is often used to refer to the consistency of Survey responses over time. Item consistency determines whether the responses for each question are consistent across constructs.

10 Test administration and scoring consistency examines the possibility of errors caused by carelessness in administration or scoring (Creswell, 1994, p. 121). Validity is the extent to which the measurements of the Survey provide the information needed to meet the study s purpose (Simon & Francis, 1998, p. 70). This definition is limited, however, to the face validity of the instrument. Content validity considers whether the questions measure the content they were intended to measure. Predictive validity examines whether the responses are able to predict a criterion measure.