There’s increasing acknowledgement of survey flaws. For example, a recent Chicago Tribune article states: “A widely cited 2015 report [by Compass] that 30 percent of Chicago founders [of tech startups] are women is not . . . valid“.
One reason the survey’s results are not accurate is that the “survey sample was not randomly selected”. Instead, survey administrators recruited respondents; they “contacted local and national partners to solicit responses to the survey through these groups’ social media and email lists”. Obviously, this recruitment “can [will?] leave out [startup founders] not connected to the groups distributing the survey”. In addition, “certain people may be more likely to respond based on availability or personal agendas”.
Other points made in the article that indicate the weaknesses of this survey are:
- “it is unclear how many local partners made an effort to find respondents”.
- “Tallying gender diversity was not the primary purpose of the surveys”. Rather, gender diversity was inferred from “dozens of data points . . . [from] a variety of sources including surveys completed by about 11,000 entrepreneurs in 40 startups . . . globally, . . . information from databases including CrunchBase and AngelList. . . . [and] 107 early-stage entrepreneurs in Chicago”.
- Compass CFO Jean-Francois Gauthier acknowledges that the sample is “limited” and “imprecise”
- With a survey “it’s . . . impossible to know how many female founders there really are”.
- “data is inaccurate. . . . flawed“
Acknowledgements of the flaws of surveys and other forms of asking, such as interviews, are increasing and indicate that confidence in survey research is waning.
For an extended discussion of the acknowledgement of the flaws of all types of survey research, see my book, The Problem with Survey Research, Chapter 5, The Problem Documented and Acknowledged, pp. 108-32.
Concerning random and non-random sampling, see my blog post, Non-random Samples and Bribes in Survey Research.