Below are my comments on Marcia Angell’s article (New York Review of Books, 12/3/15). I call attention to deficiencies perpetuated by survey researchers and by those, Angell in this instance, who rely on survey research:
Marcia Angell uses a fatally flawed method of data collection when she relies on “extensive interviews” to find out about “the actual workings” of institutional review boards (IRBs). Interviews (extensive or otherwise) do not produce reliable information about IRBs (or anything else). That’s because what people say in interviews (or in any other setting) does not necessarily correspond to what they actually do or think. It’s not that every statement is incorrect, but when you only have what people say, it’s impossible to know which, if any, of their statements is correct or incorrect. The only way to know is to check or verify what’s said with one or, preferably, two or more non-asking sources of information; such as observation and documents. Angell does not have information from non-asking sources; all she has is unreliable information; i.e., information that may, or may not, be correct.
Even those who reject the argument above and continue to believe that interviews provide reliable information need to recognize that the interviews Angell relies on can’t describe “the actual workings” of IRBs because these interviews of members of IRBs are not from a representative sample of IRBs. Sure, sixty randomly selected and, therefore, representative IRBs were identified—but only thirty-four responded! Nonresponse by almost half the sample destroys the representativeness of the interviews Angell relies on. Consequently, all her conclusions about IRBs are unfounded and some—e.g., “36 percent of IRB members have financial relationships with industry”—probably incorrect. My guess is that maybe 98 percent have conflict of interest and, maybe—given her statement that IRBs have “inherent conflict of interest since they are usually established and work within the institutions where the research is conducted”—that’s her guess also. Guesses and maybes are all that’s warranted from unrepresentative samples.
[The inability of interviews (and all other forms of asking) to produce reliable information, as well as the almost universal use of unrepresentative samples in survey research, are covered in detail in my book, The Problem with Survey Research.]