Why You Shouldn’t Use Survey Research

You shouldn’t use survey research because survey research—any procedure or instrument that asks questions—produces answers, and answers to questions are not reliable. When you only have answers to questions, it’s impossible to know whether or not one or more answers are correct or incorrect; whether or not answers do or do not correspond to what’s really going on. The only way to know is to check or verify answers with information from two or more non-asking sources of information; say, from observation, experimentation, and documents. Survey researchers—askers—don’t have information from non-asking sources and, therefore, all they have is unreliable information.

This is not to say askers never obtain answers that correspond to what’s really going on. Of course they do. Voters sometimes respond accurately in exit polls and, thereby, winners are predicted. Market researchers have received accurate information about consumer preferences and, as a consequence, corporations have sold advertised products at profit-making prices. Academic askers sometimes obtain accurate responses to questions about sexual behaviors, illicit drug use, and organizational performance. The Census Bureau, it’s generally agreed, obtains some correct information. Elected officials and corporate CEOs sometimes tell the truth about what they have done and what they intend to do. However—and as everyone knows—many exit polls have not predicted winners; not every product marketed sells as well as anticipated by survey results; people often give inaccurate answers about coitus, abortions, and heroin use, as well as workplace behaviors and organizational output; the Census Bureau does not receive accurate information about the size of minority populations, and many elected officials and corporate executives have been caught in deceptions and outright lies. This is to say, answers to questions are unreliable because when all you have are answers there’s no way to distinguish those that are correct from those that are not.

Don’t use survey research if you want to find out what’s really going on. That’s the theme of my book, The Problem with Survey Research.  In Part Six, “Proper Methods and Research Designs,” there are brief descriptions of procedures for obtaining reliable information.

Effectiveness in the Internet+ Age Dependent on Competence with Code

Our effectiveness in the Internet+ Age is dependent on our competence with code. That’s because more and more of life is online, and when online it’s all a matter of code. Consequently, we’ll do better if we understand code and we’ll do our best if we learn how to code.  

Understand Code

Everyone needs to understand code. That means, first, to know that there is such a thing as code, that code exists. In addition, we need to understand that the Internet itself, as well as everything on it, is made from code: all programs or software; everything that appears on our screens, all apps, websites, .com(s), .org(s), .edu(s), .gov(s), .net(s), images, text, colors, social networks, . . . everything!  Even some things that don’t show on the screen, such as all malware (including computer viruses, worms, and spyware) is made from code.

Code, by making all that’s online, makes, as Douglas Rushkoff points out, “[t]he . . . environments in which we all spend . . . so much of our time these days . . . where we do our work and play,” and we need to understand that these environments “have been constructed by people (or at least for people) with real agendas [interests]. They want us to believe,” Rushkoff continues, “certain ideas, spend our money on certain things, and connect to certain people in certain ways.” Code, in other words, embodies interests, and it’s important to realize that these interests may not be—and in many, if not most, cases are not—compatible with our interests.

We also need to understand that code, by making all that’s online and thus our environments, controls online behavior. Code, says William Mitchell, “control[s] when you can act, what kinds of actions you can take, and who or what you can affect by your actions.”  Code has this extensive range of control because it determines access to the Internet and, when on the Internet, code controls access to the content on it. Code–Eli Pariser informs us in his best-seller, The Filter Bubble–determines the “news we . . . consume;” what we “watch, read, and see.” Increasingly, he continues, “the power to shape the news rests in the hands of bits of code, not professional human editors.” Or, to describe this power non-metaphorically, the power to shape the news is in the hands of coders. They are the news editors of the Internet+ Age. Moreover, with each passing day, Pariser writes, more and more of “our public functions, from police databases to energy grids to schools run on code.” Even a workaday matter like a shopping trip, as pointed out in a BBC Teach post, “now relies on code to make it run smoothly.”

Consequences of Not Understanding Code

The consequences of not understanding code are appreciable. To not understand that code exists and that all that’s online is made of code is to be oblivious of a crucial component of life in the Internet+ Age. It’s not being cognizant of what’s increasingly ascent in determining what we experience and the quality of those experiences. Also, when we don’t understand code we’re “unaware,” says Rushkoff, “of the biases of the programs in which we are participating,” oblivious of a source of significant control over our lives, and ignorant of the importance of code in matters significant and mundane.

Learn How to Code

Although there are many notable advantages for understanding code, maximum effectiveness in the Internet+ Age requires that you learn how to code. That’s because code is the language of this Age. To be a literate person you need to learn how to code, how to write programs, how to design software; only then can you change the online environment coded/programmed by others and make it more compatible with your interests. Either you make code/programs to your liking or the code/programs made by others make you. In other words, “program or be programmed.” 

Another way to think about the significance of being able to code, in addition to enhancing effectiveness, is in terms of “power.” Simply put, those who are able to code have power. In Mitchell’s words, “control of code is power. . . . code . . . is the medium in which intentions are enacted and designs are realized, and it is . . . a crucial focus of political contest. Who shall write the software that increasingly structures our daily lives? What shall that software allow and proscribe? Who shall be privileged by it and who marginalized? How shall the writers of the rules be answerable?” 

Consequences of Not Learning How to Code

When you don’t know how to code, how to design software, you relegate the code/software you’re experiencing to others, thereby trusting them that their software, their programs, in Rushkoff’s words, “are really doing what you’re asking and in a way that is in your best interests.” And, he continues, “the longer you live this way, the less access you have to the knowledge that it could be any other way.” This is to say, when you don’t know how to code, you’re, as Rushkoff puts it, “at the mercy off” “coders and/or those who pay them and, moreover, you’re not aware of alternatives to what you’re experiencing. Rushkoff calls code “the steering wheel of our civilization.” So, if you don’t know how to code, he points out, “[y]ou may not know what’s going on, you may not have much of any impact on the future of our species, and you may begin to feel like technology knows more about you than you know about it.”

Postscript

There are two additional points I would like to make about code that don’t fit easily into the above discussions concerning code. The first is that code is likeable, and the second is about the time and effort required to understand and/or learn how to code.

Code is likeable. To understand code, and even more so, to know how to code is to like it. As Alexander Galloway indicates when he writes about the foundational code of the Internet, the suite of protocols named TCP/IP: “protocol is an incredibly attractive technology. . . . a technology of inclusion, . . .  openness.”. And let’s not forget, code brings us the software that increases our individual and organizational effectiveness, as well as the software that entertains us. Code is a tool of betterment–what’s not to like about that?

My second point in this postscript is that we have the time and the ability to understand code and to learn how to code. I agree with Rushkoff that it’s not “too difficult or too late” for most people in most circumstances to accomplish either or both. And as we realize the likability of code and see that increasing number of people throughout the world, including children, are understanding and learning how to code, we can become confident that we also have the time and talent to develop our competence with code.

Internet+ Effects: Twitter

There’s growing awareness that the Internet plus attendant hardware (e.g., smartphones) and software (e.g., Twitter) have fundamentally altered most, if not all, aspects of life, including our perceptions of our experiences. A case in point is Rich Lowery’s National Review article, “Twitter Deranged Our Politics.”

“Twitter,” Lowery writes, “is Exhibit A for Marshall McLuhan’s axiom that the medium is the message.” This is to say, what’s most important about Twitter is not the messages on it and their effects but rather Twitter itself and its effects. 

What are the effects of Twitter?  According to Lowery, “Twitter has made politicians dumber and cruder. . . .distorted political reality for people across the spectrum.. . . [and] helped derange our politics.”This is a misunderstanding of the nature of Twitter. It’s more accurate, I believe, to see Twitter, not as a negative force (or, conversely, as a positive force; actually, it’s both, see below) but, instead, as an accelerant—or, a la McLuhan, an “amplifier.” an “intensifier”—of what already exists. By my way of thinking:

·  Twitter has not “made politicians dumber and cruder.” Rather, Twitter clarifies and makes known to increasingly larger audiences just how dumb and crude disliked politicians are and, at the same time, clarifies and makes known to ever-larger different audiences just how smart and decent the admired ones are. Twitter spreads information fast; in this case, information about the character of America’s politicians. As a result of Twitter, we know more about our politicians than ever before.

• Twitter has not “distorted political reality for people across the spectrum.” Rather, Twitter has clarified and made known to more and more people across the spectrum the nature of the political system. As a result of Twitter, more people now understand what previously only a relatively few did (or were honest enough to say); namely that the political system is rigged, biased in favor of the wealthy and corporations. 
• Twitter has not “helped derange our politics.” Rather, Twitter has helped clarify and make known to increasingly larger audiences the nature of our politics. Lots of people now understand that our politics—and this is true for politics in any other country—is first and foremost about power or office: how to get it, how to keep it. More people realize politics is not about service, bringing us together, or doing good. Twitter gets credit for helping to promote a more accurate view of politics.

Coexistence of Opposites

Twitter helps take the gloss of the establishment narrative and helps to inform us: “There is,” says Lowery, “plenty of worthy news coverage and real-time commentary on Twitter.” At the same time, Twitter disrupts our political processes and institutions. This coexistence of opposites should not blind us to Twitter’s worthy contributions, even beneficence. 

Alchemic Askers Transform by Dropping “Say They”

Survey researchers—and, commonly, others describing or summarizing asking research (e.g., reporters and commentators)—employ alchemy to make the results of asking efforts appear more valuable than they are. In working these wizardries, survey researchers join ancient and medieval magicians who claimed to be able to transform abundant and base materials—e.g., lead—into more scarce and precious gold.

Alchemic askers transform what respondents say is going on (talk is cheap) into (more worthy) statements of what is going on by dropping the words, “say they.”  Other words dropped that perform the same magic are “reports of,” “states that,” and the like. 

In the absence of these words, what respondents say they do (objective phenomena), what they say they believe, think, or feel (subjective phenomena), and who they say they are (identification items) are transformed into statements affirming that respondents actually do what they say they do, that they really believe, think, and feel what they assert in their answers, and that they are who they say they are.

From survey researcher’s point of view, there’s good reason to drop “say they.” If askers (and those who present or analyze survey research results) inserted in their written and spoken texts “say they” or “states that” every time they reported respondents’ answers, readers and listeners would correctly conclude that askers are only reporting what people say. And, as everyone knows, what people say often does not correspond to what they do, believe, or (probably less often) who they are.

The only way to know if answers correspond to what’s really going on is to check or verify them with information/data from two or more non-asking sources; for instance, observations, documents, and/or experiments. And when you have information/data from these sources, you don’t need information/data from survey research. Give up on survey research! Why bother with what people say?

Fro more information about alchemic askers and other limitations of survey research, see, The Problem with Survey Research.

Questions Manufacture Answers

Updated March 14, 2021

A recent poll asserts that 13 percent of respondents said they believed President Joe Biden is “very responsible” for the January 6 U.S. Capital riot. This is a good example of how questions manufacture answers. To ask about Joe Biden is to produce answers about Joe Biden. If the pollsters had asked about the Pope’s responsibility for the riot, there would be answers about the Pope’s responsibility.

Here’s how I discuss this point in my book, The Problem with Survey Research: “Questions—i.e., the words and phrases that constitute them—are stimuli and reinforcers: therefore questions about employment do not generate answers or words and phrases about alcohol, sex, of health care programs. Wording of a question produces a correspondingly worded answer. In this respect, words in questions, as researcher, Lindsay Prior, points out, “manufacture” answers. An asking instrument—i.e., the words in its questions—can, says political scientist, Herbert Asher, “create,” actually bring into existence, “opinions [and other objects of inquiry] that might not otherwise be evident.” Answers aren’t “out there,” so to speak, and then asking instruments/questions find them. Asking instruments/questions make answers.”

When you want to find out what’s really going on, don’t ask. Instead, observe and use other “proper” methods and research designs.

Professional Associations, Offline Filter Bubbles, and Confirmation Bias

Professional associations are offline filter bubbles. They’re instances of the natural and, thereby, inevitable situations of being with others like us and thus experiencing and believing what others like us experience and believe. As such, professional associations instill confirmation bias, a tendency—hardly ever contravened—to use information in a way that confirms or supports already-held beliefs, interpretations, perspectives, and values. “’The American Psychological Association and American Sociological Association,’” John Staddon points out, ‘”each have more than 50 divisions. What you have,’” he continues, ‘”are little enclaves, filled with people who uncritically approve of each other’s work . . . –a collection of circular massage squads.’” The same is true for the American Political Science Association, which I belonged to for many years. APSA has 59 divisions, each filled with like-minded people experiencing confirmation bias.

Essentially all offline groups (formal and informal, large and small) that we belong to, including our family, nation-state, neighborhood, church/synagogue/mosque, country club, trade union, work-place clique, and so on—instill us with confirmation bias, just as surely as do their online counterparts, made renown by Eli Pariser in his book, The Filter Bubble.  Offline and online, there’s no escaping confirmation bias! It’s an aspect of the human condition. (Moreover—but nevertheless important, although parenthetically here—confirmation bias helps sustain the species because it’s “an efficient way to process information, protect self-esteem, and minimize cognitive dissonance.) 

Accept Implications of What Is Known

This knowledge, that filter bubbles and ensuing confirmation biases are natural and inevitable characteristics of life, encourages us to understand that beliefs, interpretations, perspectives, and values different from ours are also natural and inevitable; merely the results of different filter bubbles; that is, of different confirmation-bias-inducing families, nation-states, neighborhoods, and other offline and online groups, institutions, and processes. Who people are—what they think and do—is not their fault or choice. “If I,” says R. Buckminster Fuller, “were born and reared under the same circumstances as any other known humans, I would have behaved [and believed] much as they have.” Making disparaging comments about those different from us (Hillary Clinton’s “deplorables,” Joe Biden’s “Neanderthal thinking,” and the like) is not just pretentious; it’s also counterproductive. Rather than denouncing those with whom we disagree, we should focus on understanding the situation at hand and solving problems. Then everyone benefits.

All Societies Are Rigged

Updated March 23, 2021

All societies are rigged. That’s because every society has a structure that gives the edge to some while disadvantaging others. Even liberal societies with their deservedly celebrated tolerance cannot escape this reality. “The . . . structure of society,” Marcuse wrote in 1965, “rigs the rules of the game. Those who stand against the established system are a priori at a disadvantage, which is not removed by the toleration of their ideas, speeches, and newspapers”—and to update, or by the toleration of their blogs, Facebook posts, and Tweets.

There’s no way to unrig societies—or governments, constitutions, electoral systems or anything else. Every effort to unrig merely replaces the original structure with another structure; replaces the original rigging with another rigging. Unrigging is an illusion. The reality is: everything is rigged, always has been, always will be.  

Fractured into Three Americas and Implications for Government

One of the characteristics of our Internet+ Age is that we’re fractured into more numerous and more intense groups. Jim Vanderlei describes one aspect of this splintering, writing that America “is breaking into blue America, red America, and Trump America.” The vehemence of these three aggregations is amplified (often leading to violence) by the fact that each has its own “distinct politics, social networks, and media channels.” 

Blue America

Blue America includes the Democrat Party and its protest and often-violent and violent-inducing cadres, such as BLM, Antifa, Red Neck Revolt, and others. Also, main components of Blue America are trade unions (but less blue collar workers since 2016), elite universities and most other institutions of higher learning, urbanites, professionals, and mainstream media, such as the New York Times. 

• Leading social networks, such as Facebook and Twitter, are left-leaning components of Blue America. 
• Blue America’s media channels, it’s universally agreed, are CNN and MSNBC, with many adding PBS.   

Blue America, since the election of Joe Biden, has become “ascendant in almost every area: It won control of the House, Senate and White House; dominates traditional media; owns, controls and lives on the dominant social platforms, and has the employee-level power at Big Tech companies to force corporate decisions.”  

Red America

Red America includes the Republican Party. now split in two (another fracture within the already fractured three Americas), “starting with the relatively small Never Trumpers breaking off in 2016 and joined four years later by a new slice of establishment Republicans repulsed by President Trump’s post-election actions.” Red America’s protest cadres are anti-lockdown, anti-vaccination, and anti-abortion groups, as well as FreedomWorks, Tea Party Patriots, and other conservative activist associations. Traditionally, significant elements of Red America have been corporations, small and medium size businesses, banks, farming and rural communities, ranchers, suburbanites, investment firms, and evangelical/fundamentalist churches. However, since 2016, support from these sectors has been shifting from Red America to Trump America. 

• Red America uses the same leading social networks as Blue America; such as Facebook and Twitter.
• Red America’s media channel is Fox News.

The most basic issue facing Red America “is whether Trump America . . . eclipses the traditional Red America [including the Republican Party] in power in the coming years.”

Trump America

Trump America is, first and foremost, Donald Trump and an unknown—but probably large—percentage of the 70+ million people who voted for him in 2020. As VanderHei points out: “There’s no hard evidence yet that Trump America has shrunk significantly despite the . . . mob attack on the U.S. Capital.” Trump America’s protest and often-violent cadres include the Proud Boys, QAnon, and Oath Keepers. Trump America also includes appreciable support from corporations, small and medium size businesses, banks, farming and rural communities, ranchers, suburbanites, investment firms, and evangelical/fundamentalist churches that prior to 2016 went to Red America. In addition, Trump America has a growing number of blue collar workers, most of whom, prior to 2016, were part of Blue America, with a smaller number in Red America.   

• Trump America’s social media networks include Parler, Gab, Rumble, and Telegram.
• Trump America’s media channels are Newsmax and OANN. 

A better understanding of Trump America and the uncertainty it harbors are captured by VanderHei’s comment: “Parts of Trump America, canceled by Twitter and so many others, are severing their ties to . . . the other Americas, and basically going underground. There will be less awareness and perhaps scrutiny of what’s being said and done.”

Accept Implications of What Is Known

A major implication of the well-known fracturing of American society is that it makes impossible the give-and-take discussions, negotiations, and compromises required for the proper functioning of our government. American government—because of fractionalization, and also, and integral to fractionalization, because of filter bubbles and post-truth—is becoming increasingly dysfunctional and inefficacious. This suggests we should not continue to rely on government to solve our problems. I propose we make greater use of the Internet for problem solving.Crowdsourcing shows much promise. See also, “22 Amazing Ways to Solve Problems with Technology (Simple).”

We Can’t Think Outside the Box

Updated January 27, 2021

We can’t think outside the box because thinking is boxed in by language; specifically, by the words in language and the meaning of those words. Thinking and language/words are inextricably linked. We can’t have a thought/idea/concept that’s other than what’s (already) designated by the words in our language.

One of the clearest examples I’ve come across of trying (and, of course, failing) to think outside the box was when I was reading Robert Pirsig’s book, Zen and the Art of Motorcycle Maintenance. (This was in the late 1970s or early 1980s.) Pirsig says the basic fault of our thinking, which he articulates in the title, is that we divide reality into the subjective (Zen) and the objective (motorcycle), into spiritual and technological, into subject and object; into mind and matter. But reality, he claims, is a unified oneness; not a duality, but a singularity. Pirsig uses the word “Quality” as the overarching entity/concept/thought whereby what’s divided and dis-unified is unified. However, in order to explain what Quality is, what this oneness is, Pirsig is forced to talk about the duality/separateness that Quality unifies. The only words and thoughts that explain the unity Quality embodies are the words and thoughts that explain the duality. That’s the box our language irrevocably puts Pirsig—and all other English language/thinkers—in.    

Recently, while reading Benjamin Bratton’s 2015 book, The Stack, I found another example of trying, and failing, to think outside the box. In this instance, Bratton is concerned with what he calls/names/words the “physical” and the “virtual,” and he wants us to think about these two separate and different phenomena in a particular way: “it is only by conceivingthe design of both physical and virtualmodesof the urban interface at onceeither asconflating or complementary parts of a whole, that we can hope to intervene in their effects with consequence.” (The Stack, p. 164.)The key words for our purpose here in addition to “physical” and “virtual,” and I have underlined them, are: “conceiving[thinking] . . . of both physical and virtual modes. . . at once. . . as. . . parts of a whole.” Although we can understand what Bratton wants us to think about, the fact remains that our language does not allow us to do that; does not allow us to think of both the physical and virtual at onceand as parts of a whole. In our language, the physical and the virtual are separate; two different and separate words on the page and two different and separate ideas in our thinking. We don’t, and can’t, think of the two of them at once, but only one after the other. Nor can we think of the physical and the virtual as parts of a whole. In our language, there isn’t a word or thought that includes both; rather, the physical and the virtual are by definition/meaning in separate, different, domains.

The Only Way We Think

Thinking, like everything else, evolves incrementally. That’s the only way we think. Our thoughts at any one point in time are incremental modification of our previous thoughts. Over time, incremental changes in our initial thinking, induced by additional information and/or experiences, can lead to breakthroughs, to thoughts, ideas, theories, and understandings significantly better, more complete/accurate and, in that respect, different than the thoughts, ideas, theories, and understandings we started out with. Even Albert Einstein, who is given deserved credit for the theory of special relativity, did not develop it outside the box. Special relativity is an in-the-box culmination of his previous thinking about numerous theoretical results and empirical findings obtained by other physicists and mathematicians, as well as, according to some accounts, “ideas” of his first wife, Mileva Maric.

Think Hard and Long  

If we want to think better about whatever it is we’re thinking about, whether it’s physics, mathematics, cooking, management, engineering, coding, farming, sales, marketing, whatever! then we need to think hard and long about it. Taking a cue, again from Einstein, we can say, better thinking (his word was “genius”) is 1 percent talent and 99 percent hard work. But hard work is not enough. We have to work hard at our thinking for a long time; actually, most of the time. For certain, breakthroughs are unlikely if thinking hard occurs incidentally, or only now and then, or only at scheduled brainstorming sessions, or during annual weekend retreats. Thinking hard needs to be a defining characteristic of our life.

We can’t think outside the box, but we can think hard and long and, thereby, experience breakthroughs to better, more complete, more accurate thoughts, ideas, theories, and understandings. 

Askers Make Answers Unreliable

Askers, as is the case for instruments and settings, affect answers and, thereby, make them unreliable. Answers may be accurate and correspond to what’s really going on, or they may not. Because askers only have answers, it’s impossible for them (or anyone else) to determine which, if any, are accurate or inaccurate. The only way to know is to check or verify answers with data from non-asking sources; such as observations or experiments. Askers/survey researchers do not have data from non-asking sources; all they have is unreliable information.

Characteristics of askers that cue and induce, and thereby make unreliable, the answers they receive include their styles of behavior (e.g. asking questions rapidly, pausing, voice intonation, and so on) as well as their personal attributes; such as judgments when coding responses, experiences, competencies, ethnicity, socioeconomic features, gender, and age.

This post is from my book, The Problem with Survey Research, pp. 199-207, where documentation is provided, and on pp. 207-78 there’s an extended discussion of the characteristics and behaviors of askers.