Increasing Acknowledgement of Survey Flaws

There’s increasing acknowledgement of survey flaws.  For example, a recent Chicago Tribune article states: “A widely cited 2015 report [by Compass] that 30 percent of Chicago founders [of tech startups] are women is not . . . valid“.

One reason the survey’s results are not accurate is that the “survey sample was not randomly selected”.  Instead, survey administrators recruited respondents; they “contacted local and national partners to solicit responses to the survey through these groups’ social media and email lists”.  Obviously, this recruitment “can [will?] leave out [startup founders] not connected to the groups distributing the survey”.  In addition, “certain people may be more likely to respond based on availability or personal agendas”.

Other points made in the article that indicate the weaknesses of this survey are:

  1. “it is unclear how many local partners made an effort to find respondents”.
  2. “Tallying gender diversity was not the primary purpose of the surveys”.  Rather, gender diversity was inferred from “dozens of data points . . . [from] a variety of sources including surveys completed by about 11,000 entrepreneurs in 40 startups . . . globally,  . . . information from databases including CrunchBase and AngelList. . . . [and] 107 early-stage entrepreneurs in Chicago”.
  3. Compass CFO Jean-Francois Gauthier acknowledges that the sample is “limited”  and “imprecise”
  4. With a survey “it’s . . . impossible to know how many female founders there really are”.
  5. data is inaccurate. . . . flawed

Acknowledgements of the flaws of surveys and other forms of asking, such as interviews, are increasing and indicate that confidence in survey research is waning.

For an extended discussion of the acknowledgement of the flaws of all types of survey research, see my book, The Problem with Survey Research, Chapter 5, The Problem Documented and Acknowledged, pp. 108-32.

Concerning random and non-random sampling, see my blog post, Non-random Samples and Bribes in Survey Research.

About georgebeam

George Beam is an educator and author. The perspectives that inform his interpretations of the topics of this blog–-as well as his other writings and university courses -–are system analysis, behaviorism, and Internet effects. Specific interests include quality management, methodology, and politics. He is Associate Professor Emeritus, Department of Public Administration; Affiliated Faculty, Department of Political Science; and, previously, Head, Department of Public Administration, University of Illinois at Chicago
This entry was posted in Survey Research and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s