Survey Research Failures

There are many failures in survey research but the incorrect results of asking tend not to receive the attention they deserve.  This post is one of my efforts to call attention to the weaknesses of the asking method and, thereby, help erode confidence in survey research and make other methods–including observation and predictive modeling–more attractive.

Gallup, for instance, produced “embarrassingly incorrect results in last year’s election” (Chicago Tribune, 5/5/13, sec. 1, p. 22.).  Whereas “Gallup’s final, election eve poll showed Romney ahead by 1 percentage point, in reality, Present Barak Obama won by nearly 4 points”.  Gallup’s failure to correctly predict the election was the result of numerous aspects of survey research, including skewing by questions to screen for “likely voters” and to identify respondents’ race, and telephone polling.

Questions invariably affect/skew the answers produced by the questions.  In other words, it’s impossible to ask a question that doesn’t affect/skew the answer.  As one researcher put it, there’s no one right way to word a question.

Polling by telephone, likewise skews answers, in part, because certain categories of people (older, white, Republican) tend to have telephones more so than other categories of people (younger, minority, Democrat).  When you ask by telephone you’re going to receive more support for Republicans/Romney than for Democrats/Obama.

All four components of survey research–(1) respondents, (2) instruments (including wording of questions in polls, face-to-face interviews, etc.), (3) situations in which questions are asked and answers given (at home, in the office, on the street, etc.), and (4) survey researchers/askers themselves–singly and in combination–skew answers and make them unreliable.

To find out what’s really going on, don’t ask; instead, observe; experiment; build and test logical/predictive models; analyze contents of documents, media, etc.; and compare non-asking data.

I cover the failures of survey research and discuss observation and other “proper”–as I call them–procedures in my book, The Problem with Survey Research:http://www.amazon.com/Problem-Survey-Research-George-Beam/dp/141284603X/ref=sr_1_1?s=books&ie=UTF8&qid=1372797282&sr=1-1&keywords=the+problem+with+survey+research

A Los Angeles Times electronic version of the Tribune article with comments, links, etc. is available at: http://www.latimes.com/news/politics/la-pn-gallup-analyzes-2012-election-polls-20130604,0,5294468.story#tugs_story_display

About georgebeam

George Beam is an educator and author. The perspectives that inform his interpretations of the topics of this blog–-as well as his other writings and university courses -–are system analysis, behaviorism, and Internet effects. Specific interests include quality management, methodology, and politics. He is Associate Professor Emeritus, Department of Public Administration; Affiliated Faculty, Department of Political Science; and, previously, Head, Department of Public Administration, University of Illinois at Chicago
This entry was posted in Survey Research and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s