There are many failures in survey research but the incorrect results of asking tend not to receive the attention they deserve. This post is one of my efforts to call attention to the weaknesses of the asking method and, thereby, help erode confidence in survey research and make other methods–including observation and predictive modeling–more attractive.
Gallup, for instance, produced “embarrassingly incorrect results in last year’s election” (Chicago Tribune, 5/5/13, sec. 1, p. 22.). Whereas “Gallup’s final, election eve poll showed Romney ahead by 1 percentage point, in reality, Present Barak Obama won by nearly 4 points”. Gallup’s failure to correctly predict the election was the result of numerous aspects of survey research, including skewing by questions to screen for “likely voters” and to identify respondents’ race, and telephone polling.
Questions invariably affect/skew the answers produced by the questions. In other words, it’s impossible to ask a question that doesn’t affect/skew the answer. As one researcher put it, there’s no one right way to word a question.
Polling by telephone, likewise skews answers, in part, because certain categories of people (older, white, Republican) tend to have telephones more so than other categories of people (younger, minority, Democrat). When you ask by telephone you’re going to receive more support for Republicans/Romney than for Democrats/Obama.
All four components of survey research–(1) respondents, (2) instruments (including wording of questions in polls, face-to-face interviews, etc.), (3) situations in which questions are asked and answers given (at home, in the office, on the street, etc.), and (4) survey researchers/askers themselves–singly and in combination–skew answers and make them unreliable.
To find out what’s really going on, don’t ask; instead, observe; experiment; build and test logical/predictive models; analyze contents of documents, media, etc.; and compare non-asking data.
I cover the failures of survey research and discuss observation and other “proper”–as I call them–procedures in my book, The Problem with Survey Research.