Question and Questionnaire Design

1 Chapter 9 Question and Questionnaire DesignJon A. Krosnick and Stanley PresserThe heart of a survey is its Questionnaire . Drawing a sample, hiring, and traininginterviewers and supervisors, programming computers, and other preparatory workis all in service of the conversation that takes place between researchers andrespondents. survey results depend crucially on the Questionnaire that scripts thisconversation (irrespective of how the conversation is mediated, , by aninterviewer or a computer). To minimize response errors, questionnaires should becrafted in accordance with best about best practices stem from experience and common lore,on the one hand, and methodological research, on the other. In this chapter, we firstoffer recommendations about optimal Questionnaire Design based on conventionalwisdom (focusing mainly on the words used in questions), and then make furtherrecommendations based on a review of the methodological research (focusing mainlyon the structural features of questions).

2 We begin our examination of the methodological literature by consideringopen versus closed questions, a difference especially relevant to three types ofmeasurement: (1) asking for choices among nominal categories ( , What is themost important problem facing the country? ), (2) ascertaining numeric quantities( , How many hours did you watch television last week? ), and (3) testing factualknowledge ( , Who is Joseph Biden? ).Next, we discuss the Design of rating scales. We review the literature on theoptimal number of scale points, consider whether some or all scale points should belabeled with words and/or numbers, and examine the problem of acquiescenceresponse bias and methods for avoiding it. We then turn to the impact of responseoption order, outlining how it varies depending on whether categories are nominal orordinal and whether they are presented visually or that, we assess whether to offer don t know or no-opinion among aquestion s explicit response options.

3 Next we discuss social desirability response biasHandbook of survey Research, Second EditionCopyrightr2010 by Emerald Group Publishing LimitedAll rights of reproduction in any form reservedISBN: 978-1-84855-224-1(a form of motivated misreporting) and recall bias (a form of unmotivatedmisreporting), and recommend ways to minimize each. Following that, we considerthe ordering of questions within a Questionnaire and then discuss methods for testingand evaluating questions and questionnaires. Finally, we offer two more generalrecommendations to guide Questionnaire Conventional WisdomHundreds of methodology textbooks have offered various versions of conventionalwisdom about optimal Question Design . The most valuable advice in this commonwisdom can be summarized as follows:1.

4 Use simple, familiar words (avoid technical terms, jargon, and slang);2. Use simple syntax;3. Avoid words with ambiguous meanings, , aim for wording that all respondentswill interpret in the same way;4. Strive for wording that is specific and concrete (as opposed to general andabstract);5. Make response options exhaustive and mutually exclusive;6. Avoid leading or loaded questions that push respondents toward an answer;7. Ask about one thing at a time (avoid double-barreled questions); and8. Avoid questions with single or double wisdom also contains advice about how to optimize questionorder:1. Early questions should be easy and pleasant to answer, and should build rapportbetween the respondent and the Questions at the very beginning of a Questionnaire should explicitly addressthe topic of the survey , as it was described to the respondent prior to Questions on the same topic should be grouped Questions on the same topic should proceed from general to Questions on sensitive topics that might make respondents uncomfortable shouldbe placed at the end of the Filter questions should be included, to avoid asking respondents questions that donot apply to , conventional wisdom recommends pretesting questionnaires.

5 Though ithas little to say about how this is best together these recommendations are of great value, but there is even moreto be learned from the results of methodological A. Krosnick and Stanley Optimizing versus SatisficingThere is widespread agreement about the cognitive processes involved in answeringquestions optimally ( , Cannell, Miller, & Oksenberg, 1981; Schwarz & Strack, 1985;Tourangeau & Rasinski, 1988). Specifically, respondents are presumed to execute eachof four steps. First, they must interpret the Question and deduce its intent. Next, theymust search their memories for relevant information, and then integrate whateverinformation comes to mind into a single judgment. Finally, they must translate thejudgment into a response, by selecting one of the alternatives offered by the of these steps can be quite complex, involving considerable cognitivework (see Tourangeau & Bradburn, this volume).

6 A wide variety of motives mayencourage respondents to do this work, including desires for self-expression,interpersonal response, intellectual challenge, self-understanding, altruism, oremotional catharsis (see Warwick & Lininger, 1975, pp. 185 187). Effort can alsobe motivated by the desire to assist the survey sponsor, , to help employersimprove working conditions, businesses Design better products, or governmentsmake better-informed policy. To the extent that such motives inspire a respondentto perform the necessary cognitive tasks in a thorough and unbiased manner, therespondent may be said to much as we hope all respondents will optimize throughout a Questionnaire ,this is often an unrealistic expectation. Some people may agree to complete aquestionnaire as result of a relatively automatic compliance process (see, ,Cialdini, 1993) or because they are required to do so.

7 Thus, they may agree merely toprovide answers, with no intrinsic motivation to make the answers of high respondents may satisfy whatever desires motivated them to participate afteranswering a first set of questions, and become fatigued, disinterested, or distracted asa Questionnaire progresses than expend the effort necessary to provide optimal answers, respondentsmay take subtle or dramatic shortcuts. In the former case, respondents may simplybe less thorough in comprehension, retrieval, judgment, and response may be less thoughtful about a Question s meaning; search their memories lesscomprehensively; integrate retrieved information less carefully; or select a responsechoice less precisely. All four steps are executed, but less diligently than whenoptimizing occurs.

8 Instead of attempting the most accurate answers, respondentssettle for merely satisfactory answers. The first answer a respondent considers thatseems acceptable is the one offered. This response behavior might be termed weaksatisficing(Krosnick, 1991, borrowing the term from Simon, 1957).A more dramatic shortcut is to skip the retrieval and judgment steps is, respondents may interpret each Question superficially and select what theybelieve will appear to be a reasonable answer. The answer is selected withoutreference to any internal psychological cues specifically relevant to the attitude,belief, or event of interest. Instead, the respondent may look to the wording of thequestion for a cue, pointing to a response that can be easily selected and easilydefended if necessary.

9 If no such cue is present, the respondent may select an answercompletely arbitrarily. This process might be termedstrong and Questionnaire Design265It is useful to think of optimizing and strong satisficing as the two ends of acontinuum indicating the degrees of thoroughness with which the four response stepsare performed. The optimizing end of the continuum involves complete and effortfulexecution of all four steps. The strong satisficing end involves little effort in theinterpretation and answer reporting steps and no retrieval or integration at between are intermediate likelihood of satisficing is thought to be determined by three major factors:task difficulty, respondent ability, and respondent motivation (Krosnick, 1991).

10 Taskdifficulty is a function of both Question -specific attributes ( , the difficulty ofinterpreting a Question and of retrieving and manipulating the requested informa-tion) and attributes of the Questionnaire s administration ( , the pace at which aninterviewer reads the questions and the presence of distracting events). Ability isshaped by the extent to which respondents are adept at performing complex mentaloperations, practiced at thinking about the topic of a particular Question , andequipped with preformulated judgments on the issue in Question . Motivation isinfluenced by need for cognition (Cacioppo, Petty, Feinstein, & Jarvis, 1996), thedegree to which the topic of a Question is personally important, beliefs about whetherthe survey will have useful consequences, respondent fatigue, and aspects ofquestionnaire administration (such as interviewer behavior)