Question Formatting Makes Answers Unreliable

Question formatting makes answers unreliable because the physical features of instruments (e.g., instrument length), structures of questions (e.g., open-ended, and fixed response questions), and patterns in which questions are related to each other (e.g., placing questions about personal matters, such as gender or income, before of after questions about the topic or issue being investigated) affect both response rates and contents of answers.  As you’ll notice in the quoted material below (from a university survey research center newsletter), survey researchers know that formatting skews answers, and you’ll also notice that they do not–because they cannot!– indicate how to format questions so that answers are not skewed/biased.  No matter how questions are asked, the question asked affects/biases the answer given.  Because you cannot obtain  unaffected/unbiased/reliable answers, you shouldn’t ask.  Don’t ask!

Survey researchers continue to ask even though they know they’re receiving biased/skewed answers because… because…. well, because they’re addicted to asking!

For additional information about affects of question formats and the addiction to asking, see my book, The Problem with Survey Research, especially Chapter 6, “Shaky Instruments” and Chapter 11, “Addicted Askers”.


“Standardized Reading and Question Formatting

The way questions are presented for an interviewer to read helps achieve the goal of standardized questionnaire administration. Standardized reading can be an ongoing challenge when multiple interviewers are working on a study, and attending to the details of question formatting and writing helps in this process.

• Questions should be scripted so that interviewers are not tempted (or forced) to add anything to make a question sound complete. Questions that seem complete on paper may not be readable aloud. An absence of question stems (Would you say…) can lead to different interviewer readings. One interviewer may add a stem, but another may not. Similarly, interviewers may read response categories differently in the absence of punctuation such as commas.

• Complicated question formats, while they can save space on a screen, often leave out things (such as question stems, punctuation, or even words) that make for good reading.

• Use standard formatting for emphasis, text that should be read vs. not read, and for notation for acceptable readings in repetitive question series. Interviewers should understand the conventions that are used to express all of these things. Inconsistent question formatting or notation for emphasis can lead to different interviewer readings.

• Avoid parentheticals (words that clarify other words); they usually are not readable aloud.

• If interviewers are allowed to provide definitions, or if specific instructions are required for data entry or coding, they should be provided on screen, rather than left to interviewer memory.

• Providing scripted transitions between sections can help interviewers avoid the temptation to add words in an effort to be “conversational.”

• The scripting of one mode (such as Web) will likely not transfer seamlessly to interviewer administration.

Read-throughs with interviewers and mock interview practice can reveal surprising things about how interviewers see questions and can be helpful in identifying shortcomings before a questionnaire is fielded.”

About georgebeam

George Beam is an educator and author. The perspectives that inform his interpretations of the topics of this blog–-as well as his other writings and university courses -–are system analysis, behaviorism, and Internet effects. Specific interests include quality management, methodology, and politics. He is Associate Professor Emeritus, Department of Public Administration; Affiliated Faculty, Department of Political Science; and, previously, Head, Department of Public Administration, University of Illinois at Chicago
This entry was posted in Survey Research and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s