Asking Settings Make Answers Unreliable

Settings in which questions are asked and answers given make answers unreliable; that is, answers that may, or may not, be accurate. This happens because components in the two types of settings (societal and immediate, discussed below) skew answers, and since askers/survey researchers only have answers to their questions, it’s impossible for them (or anyone else) to determine which, if any, are either accurate or inaccurate. The only way to know is to check or verify answers with data/information from non-asking sources; for example, from observations or experiments. Askers do not have this type of data; all they have is unreliable information.

There are two types of asking settings: societal and immediate.
• Societal settings are cultures experienced by respondents, and that includes social values, illusions/myths/religions, and descriptions and interpretations of political and economic systems. Also included in respondents’ societal settings are their positions in society—e.g., socioeconomic, organizational, familial, and so on—and the norms and beliefs associated with these positions. Via socialization by parents, schools, colleagues, governments, and advertising, respondents take as their own the societal preferences, priorities, and outlooks they experience and answer accordingly.
• Immediate settings are specific places where asking and answering occur: for example, respondents’ homes and workplaces (via phone and Internet), on streets outside polling places (entrance and exit polls), schools, doctors’ offices, and so on.
Both types of settings (societal and immediate) have powerful attributes or components that affect answers. Characteristics of, and forces within, societal settings—such as cultural values, laws, and media—bias answers, as do aspects of immediate settings, including third parties, gender of interviewer, and so on. Essentially every component of every asking setting “contaminates;” forcing respondents to say what’s compatible with each setting.

This post is from my book, The Problem with Survey Research, p. 165, where sources are cited and where, on pp. 165-95, you can find an expanded discussion of how asking settings make answers unreliable.