The Problem with Survey Research Documented

The Problem with survey research is documented throughout the social sciences—e.g., in anthropology, sociology, and psychology—and in management studies, political polling, and elsewhere. Research demonstrates that respondents give answers that do not correspond to data from documents and observations, are contradictory and/or inconsistent, and biased by a host of factors, such as ethnicity and gender of askers, question topic, and so on. Distortions between respondents’ answers and their behavior range from 4 percent to 89 percent, with 40-50 percent not uncommon.

Inaccurate responses are given to every topic, including, of course, murder and lesser illegalities. Questions asked about activities that can lead to imprisonment and, sometimes, death, vividly indicate the unreliability of answers. First—and as everyone knows—many who, according to the facts, have murdered and committed other crimes say they are innocence. On the other hand, individuals sometimes say they’ve done what evidence indicates they haven’t; they falsely confess. Actually,there are so many false confessions that increasing numbers of the general public, reporters, lawyers, and judges have come to doubt the reliability of confessions.

Other topics for which there is documented evidence that truth-telling or accuracy is sometimes, and often, absent include: home address, age, number of rooms in dwelling, and similar sorts of information asked for by the Census Bureau, types and amounts of food eaten and wasted, extent of education and formal schooling, cheating on examinations, visits to doctors’ offices, illnesses and hospitalizations, surgical procedures experienced (including abortions), treatment for illicit drug usage, purchase and consumption of alcoholic beverages, smoking, use of contraceptives, divorce, voter registration, voting behavior, driving record, work history (including absenteeism, previous pay/salary and changes in pay/salary), classroom and workplace behavior (e.g., activity at the office, on the shop floor, hours of the road driving an eighteen-wheeler, and hiring the physically challenged), income and/or salary, characteristic of, and changes in, government programs, objective measures of well-being, behavior while in the military, attendance at church, attendance at trade union meetings, personnel debt, bank account data, involvement in bankruptcy, business investments, cash loans, ATM withdrawals, contributions to charities, behavior of other people, and of special import to survey researchers, receiving and returning questionnaires.

There are two take-a-ways from the documentation of the Problem with survey research:

·     Don’t ask to find out what’s really going on. Instead, observe or use other non-asking methods and research designs, such as experimentation. 

·     If you insist on asking, then you should check or verify your answers with observations, experiments, and the like. Never rely on answers—they’re unreliable! 

For a version of this post with supporting references, see The Problem with Survey Research, pp. 107-08. 

The Problem with survey research

The Problem with survey research is that answers to questions are not reliable. When you only have answers to questions it’s impossible to know if the answers correspond to what’s really going on. The only way to know if an answer is correct or accurate is to check or verify it with information from two, or preferably three, non-asking sources of information; say, from observations, experiments, and/or documents. Those who rely on the asking method do not have information from non-asking sources and, therefore, are not able to discern which, if any, answer is correct or incorrect. All survey researchers have is unreliable information.

If you want to find out what’s really going on, don’t ask. That’s the theme of my book, The Problem with Survey Research.

Why You Shouldn’t Use Survey Research

You shouldn’t use survey research because survey research—any procedure or instrument that asks questions—produces answers, and answers to questions are not reliable. When you only have answers to questions, it’s impossible to know whether or not one or more answers are correct or incorrect; whether or not answers do or do not correspond to what’s really going on. The only way to know is to check or verify answers with information from two or more non-asking sources of information; say, from observation, experimentation, and documents. Survey researchers—askers—don’t have information from non-asking sources and, therefore, all they have is unreliable information.

This is not to say askers never obtain answers that correspond to what’s really going on. Of course they do. Voters sometimes respond accurately in exit polls and, thereby, winners are predicted. Market researchers have received accurate information about consumer preferences and, as a consequence, corporations have sold advertised products at profit-making prices. Academic askers sometimes obtain accurate responses to questions about sexual behaviors, illicit drug use, and organizational performance. The Census Bureau, it’s generally agreed, obtains some correct information. Elected officials and corporate CEOs sometimes tell the truth about what they have done and what they intend to do. However—and as everyone knows—many exit polls have not predicted winners; not every product marketed sells as well as anticipated by survey results; people often give inaccurate answers about coitus, abortions, and heroin use, as well as workplace behaviors and organizational output; the Census Bureau does not receive accurate information about the size of minority populations, and many elected officials and corporate executives have been caught in deceptions and outright lies. This is to say, answers to questions are unreliable because when all you have are answers there’s no way to distinguish those that are correct from those that are not.

Don’t use survey research if you want to find out what’s really going on. That’s the theme of my book, The Problem with Survey Research.  In Part Six, “Proper Methods and Research Designs,” there are brief descriptions of procedures for obtaining reliable information.

Effectiveness in the Internet+ Age Dependent on Competence with Code

Our effectiveness in the Internet+ Age is dependent on our competence with code. That’s because more and more of life is online, and when online it’s all a matter of code. Consequently, we’ll do better if we understand code and we’ll do our best if we learn how to code.  

Understand Code

Everyone needs to understand code. That means, first, to know that there is such a thing as code, that code exists. In addition, we need to understand that the Internet itself, as well as everything on it, is made from code: all programs or software; everything that appears on our screens, all apps, websites, .com(s), .org(s), .edu(s), .gov(s), .net(s), images, text, colors, social networks, . . . everything!  Even some things that don’t show on the screen, such as all malware (including computer viruses, worms, and spyware) is made from code.

Code, by making all that’s online, makes, as Douglas Rushkoff points out, “[t]he . . . environments in which we all spend . . . so much of our time these days . . . where we do our work and play,” and we need to understand that these environments “have been constructed by people (or at least for people) with real agendas [interests]. They want us to believe,” Rushkoff continues, “certain ideas, spend our money on certain things, and connect to certain people in certain ways.” Code, in other words, embodies interests, and it’s important to realize that these interests may not be—and in many, if not most, cases are not—compatible with our interests.

We also need to understand that code, by making all that’s online and thus our environments, controls online behavior. Code, says William Mitchell, “control[s] when you can act, what kinds of actions you can take, and who or what you can affect by your actions.”  Code has this extensive range of control because it determines access to the Internet and, when on the Internet, code controls access to the content on it. Code–Eli Pariser informs us in his best-seller, The Filter Bubble–determines the “news we . . . consume;” what we “watch, read, and see.” Increasingly, he continues, “the power to shape the news rests in the hands of bits of code, not professional human editors.” Or, to describe this power non-metaphorically, the power to shape the news is in the hands of coders. They are the news editors of the Internet+ Age. Moreover, with each passing day, Pariser writes, more and more of “our public functions, from police databases to energy grids to schools run on code.” Even a workaday matter like a shopping trip, as pointed out in a BBC Teach post, “now relies on code to make it run smoothly.”

Consequences of Not Understanding Code

The consequences of not understanding code are appreciable. To not understand that code exists and that all that’s online is made of code is to be oblivious of a crucial component of life in the Internet+ Age. It’s not being cognizant of what’s increasingly ascent in determining what we experience and the quality of those experiences. Also, when we don’t understand code we’re “unaware,” says Rushkoff, “of the biases of the programs in which we are participating,” oblivious of a source of significant control over our lives, and ignorant of the importance of code in matters significant and mundane.

Learn How to Code

Although there are many notable advantages for understanding code, maximum effectiveness in the Internet+ Age requires that you learn how to code. That’s because code is the language of this Age. To be a literate person you need to learn how to code, how to write programs, how to design software; only then can you change the online environment coded/programmed by others and make it more compatible with your interests. Either you make code/programs to your liking or the code/programs made by others make you. In other words, “program or be programmed.” 

Another way to think about the significance of being able to code, in addition to enhancing effectiveness, is in terms of “power.” Simply put, those who are able to code have power. In Mitchell’s words, “control of code is power. . . . code . . . is the medium in which intentions are enacted and designs are realized, and it is . . . a crucial focus of political contest. Who shall write the software that increasingly structures our daily lives? What shall that software allow and proscribe? Who shall be privileged by it and who marginalized? How shall the writers of the rules be answerable?” 

Consequences of Not Learning How to Code

When you don’t know how to code, how to design software, you relegate the code/software you’re experiencing to others, thereby trusting them that their software, their programs, in Rushkoff’s words, “are really doing what you’re asking and in a way that is in your best interests.” And, he continues, “the longer you live this way, the less access you have to the knowledge that it could be any other way.” This is to say, when you don’t know how to code, you’re, as Rushkoff puts it, “at the mercy off” “coders and/or those who pay them and, moreover, you’re not aware of alternatives to what you’re experiencing. Rushkoff calls code “the steering wheel of our civilization.” So, if you don’t know how to code, he points out, “[y]ou may not know what’s going on, you may not have much of any impact on the future of our species, and you may begin to feel like technology knows more about you than you know about it.”

Postscript

There are two additional points I would like to make about code that don’t fit easily into the above discussions concerning code. The first is that code is likeable, and the second is about the time and effort required to understand and/or learn how to code.

Code is likeable. To understand code, and even more so, to know how to code is to like it. As Alexander Galloway indicates when he writes about the foundational code of the Internet, the suite of protocols named TCP/IP: “protocol is an incredibly attractive technology. . . . a technology of inclusion, . . .  openness.”. And let’s not forget, code brings us the software that increases our individual and organizational effectiveness, as well as the software that entertains us. Code is a tool of betterment–what’s not to like about that?

My second point in this postscript is that we have the time and the ability to understand code and to learn how to code. I agree with Rushkoff that it’s not “too difficult or too late” for most people in most circumstances to accomplish either or both. And as we realize the likability of code and see that increasing number of people throughout the world, including children, are understanding and learning how to code, we can become confident that we also have the time and talent to develop our competence with code.

Internet+ Effects: Twitter

Updated October 17, 2021

There’s growing awareness that the Internet plus attendant hardware (e.g., smartphones) and software (e.g., Twitter) have fundamentally altered most, if not all, aspects of life, including our perceptions of our experiences. A case in point is Rich Lowery’s National Review article, “Twitter Deranged Our Politics.”

“Twitter,” Lowery writes, “is Exhibit A for Marshall McLuhan’s axiom that the medium is the message.” This is to say, what’s most important about Twitter is not the messages on it and their effects but rather Twitter itself and its effects. 

What are the effects of Twitter?  According to Lowery, “Twitter has made politicians dumber and cruder. . . .distorted political reality for people across the spectrum.. . . [and] helped derange our politics.” This is a misunderstanding of the nature of Twitter. It’s more accurate, I believe, to see Twitter, not as a negative force (or, conversely, as a positive force; actually, it’s both, see below) but, instead, as an accelerate—or, a la McLuhan, an “amplifier,” an “intensifier”—of what already exists. By my way of thinking:

·  Twitter has not “made politicians dumber and cruder.” Rather, Twitter clarifies and makes known to increasingly larger audiences just how dumb and crude disliked politicians are and, at the same time, clarifies and makes known to ever-larger different audiences just how smart and decent the admired ones are. Twitter spreads information fast; in this case, information about the character of America’s politicians. As a result of Twitter, we know more about our politicians than ever before.

• Twitter has not “distorted political reality for people across the spectrum.” Rather, Twitter has clarified and made known to more and more people the nature of the political system. As a result of Twitter, more people now understand what previously only a relatively few did (or were honest enough to say); namely, that the political system is rigged, biased in favor of the wealthy and corporations. 
• Twitter has not “helped derange our politics.” Rather, Twitter has helped clarify and make known to increasingly larger audiences the nature of our politics. Lots of people now understand that our politics—and this is true for politics in any other country—is first and foremost about power or office: how to get it, how to keep it. More people realize politics is not about service, bringing us together, or doing good. Twitter gets credit for helping to promote a more accurate view of politics.

Summary

Twitter helps inform us and thereby some of the gloss is taken off the establishment narrative. The disruptions to our institutions and processes caused by this growing awareness of the way things actually are should not blind us to Twitter’s worthy contributions, even beneficence. 

Survey Researchers Transform by Dropping “Say They”

Updated August 26, 2021

Survey researchers—and, commonly, others describing or summarizing asking research (e.g., reporters and commentators)—employ alchemy to make the results of asking efforts appear more valuable than they are. In working these wizardries, survey researchers join ancient and medieval magicians who claimed to be able to transform abundant and base materials—e.g., lead—into more scarce and precious gold.

Alchemic askers transform what respondents say is going on (talk is cheap) into (more worthy) statements of what is going on by dropping the words, “say they.”  Other words dropped that perform the same alchemy/magic are “reports of,” “states that,” and the like. 

In the absence of these words, what respondents say they do (objective phenomena), what they say they believe, think, or feel (subjective phenomena), and who they say they are (identification items) are transformed into statements affirming that respondents actually do what they say they do, that they really believe, think, and feel what they assert in their answers, and that they are who they say they are.

From survey researcher’s point of view, there’s good reason to drop “say they.” If askers (and those who present or analyze survey research results) inserted in their written and spoken texts “say they” or “states that” every time they reported respondents’ answers, readers and listeners would correctly conclude that askers are only reporting what people say. And, as everyone knows, what people say often does not correspond to what they do, believe, or (probably less often) who they are.

The only way to know if answers correspond to what’s really going on is to check or verify them with information/data from two or more non-asking sources; for instance, observations, documents, and/or experiments. And when you have information/data from these sources, you don’t need information/data from survey research. Give up on survey research! Why bother with what people say?

Fro more information about alchemic askers and other limitations of survey research, see, The Problem with Survey Research.

Questions Manufacture Answers

Updated March 14, 2021

A recent poll asserts that 13 percent of respondents said they believed President Joe Biden is “very responsible” for the January 6 U.S. Capital riot. This is a good example of how questions manufacture answers. To ask about Joe Biden is to produce answers about Joe Biden. If the pollsters had asked about the Pope’s responsibility for the riot, there would be answers about the Pope’s responsibility.

Here’s how I discuss this point in my book, The Problem with Survey Research: “Questions—i.e., the words and phrases that constitute them—are stimuli and reinforcers: therefore questions about employment do not generate answers or words and phrases about alcohol, sex, of health care programs. Wording of a question produces a correspondingly worded answer. In this respect, words in questions, as researcher, Lindsay Prior, points out, “manufacture” answers. An asking instrument—i.e., the words in its questions—can, says political scientist, Herbert Asher, “create,” actually bring into existence, “opinions [and other objects of inquiry] that might not otherwise be evident.” Answers aren’t “out there,” so to speak, and then asking instruments/questions find them. Asking instruments/questions make answers.”

When you want to find out what’s really going on, don’t ask. Instead, observe and use other “proper” methods and research designs.

Professional Associations, Offline Filter Bubbles, and Confirmation Bias

Professional associations are offline filter bubbles. They’re instances of the natural and, thereby, inevitable situations of being with others like us and thus experiencing and believing what others like us experience and believe. As such, professional associations instill confirmation bias, a tendency—hardly ever contravened—to use information in a way that confirms or supports already-held beliefs, interpretations, perspectives, and values. “’The American Psychological Association and American Sociological Association,’” John Staddon points out, ‘”each have more than 50 divisions. What you have,’” he continues, ‘”are little enclaves, filled with people who uncritically approve of each other’s work . . . –a collection of circular massage squads.’” The same is true for the American Political Science Association, which I belonged to for many years. APSA has 59 divisions, each filled with like-minded people experiencing confirmation bias.

Essentially all offline groups (formal and informal, large and small) that we belong to, including our family, nation-state, neighborhood, church/synagogue/mosque, country club, trade union, work-place clique, and so on—instill us with confirmation bias, just as surely as do their online counterparts, made renown by Eli Pariser in his book, The Filter Bubble.  Offline and online, there’s no escaping confirmation bias! It’s an aspect of the human condition. (Moreover—but nevertheless important, although parenthetically here—confirmation bias helps sustain the species because it’s “an efficient way to process information, protect self-esteem, and minimize cognitive dissonance.) 

Accept Implications of What Is Known

This knowledge, that filter bubbles and ensuing confirmation biases are natural and inevitable characteristics of life, encourages us to understand that beliefs, interpretations, perspectives, and values different from ours are also natural and inevitable; merely the results of different filter bubbles; that is, of different confirmation-bias-inducing families, nation-states, neighborhoods, and other offline and online groups, institutions, and processes. Who people are—what they think and do—is not their fault or choice. “If I,” says R. Buckminster Fuller, “were born and reared under the same circumstances as any other known humans, I would have behaved [and believed] much as they have.” Making disparaging comments about those different from us (Hillary Clinton’s “deplorables,” Joe Biden’s “Neanderthal thinking,” and the like) is not just pretentious; it’s also counterproductive. Rather than denouncing those with whom we disagree, we should focus on understanding the situation at hand and solving problems. Then everyone benefits.

All Societies Are Rigged

Updated March 23, 2021

All societies are rigged. That’s because every society has a structure that gives the edge to some while disadvantaging others. Even liberal societies with their deservedly celebrated tolerance cannot escape this reality. “The . . . structure of society,” Marcuse wrote in 1965, “rigs the rules of the game. Those who stand against the established system are a priori at a disadvantage, which is not removed by the toleration of their ideas, speeches, and newspapers”—and to update, or by the toleration of their blogs, Facebook posts, and Tweets.

There’s no way to unrig societies—or governments, constitutions, electoral systems or anything else. Every effort to unrig merely replaces the original structure with another structure; replaces the original rigging with another rigging. Unrigging is an illusion. The reality is: everything is rigged, always has been, always will be.  

Fractured into Three Americas and Implications for Government

One of the characteristics of our Internet+ Age is that we’re fractured into more numerous and more intense groups. Jim Vanderlei describes one aspect of this splintering, writing that America “is breaking into blue America, red America, and Trump America.” The vehemence of these three aggregations is amplified (often leading to violence) by the fact that each has its own “distinct politics, social networks, and media channels.” 

Blue America

Blue America includes the Democrat Party and its protest and often-violent and violent-inducing cadres, such as BLM, Antifa, Red Neck Revolt, and others. Also, main components of Blue America are trade unions (but less blue collar workers since 2016), elite universities and most other institutions of higher learning, urbanites, professionals, and mainstream media, such as the New York Times. 

• Leading social networks, such as Facebook and Twitter, are left-leaning components of Blue America. 
• Blue America’s media channels, it’s universally agreed, are CNN and MSNBC, with many adding PBS.   

Blue America, since the election of Joe Biden, has become “ascendant in almost every area: It won control of the House, Senate and White House; dominates traditional media; owns, controls and lives on the dominant social platforms, and has the employee-level power at Big Tech companies to force corporate decisions.”  

Red America

Red America includes the Republican Party. now split in two (another fracture within the already fractured three Americas), “starting with the relatively small Never Trumpers breaking off in 2016 and joined four years later by a new slice of establishment Republicans repulsed by President Trump’s post-election actions.” Red America’s protest cadres are anti-lockdown, anti-vaccination, and anti-abortion groups, as well as FreedomWorks, Tea Party Patriots, and other conservative activist associations. Traditionally, significant elements of Red America have been corporations, small and medium size businesses, banks, farming and rural communities, ranchers, suburbanites, investment firms, and evangelical/fundamentalist churches. However, since 2016, support from these sectors has been shifting from Red America to Trump America. 

• Red America uses the same leading social networks as Blue America; such as Facebook and Twitter.
• Red America’s media channel is Fox News.

The most basic issue facing Red America “is whether Trump America . . . eclipses the traditional Red America [including the Republican Party] in power in the coming years.”

Trump America

Trump America is, first and foremost, Donald Trump and an unknown—but probably large—percentage of the 70+ million people who voted for him in 2020. As VanderHei points out: “There’s no hard evidence yet that Trump America has shrunk significantly despite the . . . mob attack on the U.S. Capital.” Trump America’s protest and often-violent cadres include the Proud Boys, QAnon, and Oath Keepers. Trump America also includes appreciable support from corporations, small and medium size businesses, banks, farming and rural communities, ranchers, suburbanites, investment firms, and evangelical/fundamentalist churches that prior to 2016 went to Red America. In addition, Trump America has a growing number of blue collar workers, most of whom, prior to 2016, were part of Blue America, with a smaller number in Red America.   

• Trump America’s social media networks include Parler, Gab, Rumble, and Telegram.
• Trump America’s media channels are Newsmax and OANN. 

A better understanding of Trump America and the uncertainty it harbors are captured by VanderHei’s comment: “Parts of Trump America, canceled by Twitter and so many others, are severing their ties to . . . the other Americas, and basically going underground. There will be less awareness and perhaps scrutiny of what’s being said and done.”

Accept Implications of What Is Known

A major implication of the well-known fracturing of American society is that it makes impossible the give-and-take discussions, negotiations, and compromises required for the proper functioning of our government. American government—because of fractionalization, and also, and integral to fractionalization, because of filter bubbles and post-truth—is becoming increasingly dysfunctional and inefficacious. This suggests we should not continue to rely on government to solve our problems. I propose we make greater use of the Internet for problem solving.Crowdsourcing shows much promise. See also, “22 Amazing Ways to Solve Problems with Technology (Simple).”