Translate Survey Research Questions into Statements that Can be Verified

One way to reduce reliance on survey research is to translate survey research questions into statements that can be checked or verified by observation, content analysis, and other–as I call them in Part 6 of The Problem with Survey Research–“proper” methods of data collection and proper research designs.   That’s what I did in response to an invitation from the World Bank’s, iChallenge, concerning indicators of the performance of public management systems.  I translated 7 survey research questions into statements that can be checked or verified by content analysis of documents and reports.  (For example–and as you can see in 1.–asking graduates how they “view” a public sector career is translated into statements of employment that can be checked or verified by government documents/records.)

1. “How do graduates of the country’s most elite university(ies) view a public sector career?” becomes: Percentage of graduates of the country’s most elite university(ies) who, within one year, are employed at (level/function; e.g., executive level, street-level, supervisory function, etc.) in the public sector.

2. “How do members of the educated middle class who are not in a position to attend the most elite universities view a public sector career?” becomes: Percentage of public sector personnel at (level/function; e.g., executive level, street-level, supervisory function, etc.) who are from the educated middle class.

3. “When deciding how to implement policies in individual cases, public sector employees treat some groups in society unfairly?” becomes: Percentage of street-level public sector personnel who make the same decisions in individual cases for citizens from different groups in society.

4. “Public sector employees strive to follow rules.” becomes: Percent of public sector personnel at (level/function; e.g., executive level, street-level, supervisory function, etc.) who follow rules (specify rules/procedures; e.g., rules/procedures in procurement, hiring, etc.).

5. “Public sector employees strive to help citizens.” becomes: Percent of public sector personnel at (level/function; e.g., executive level, street-level, supervisory function, etc.) who by (specify activity/behavior, which varies by level/function; e.g., content of managers’ telephone conversations and/or emails, executives’ decisions that promote access to government documents, etc.) help citizens.

6. “Public sector employees strive to implement the policies decided by the top political leadership.” becomes: Percent of public sector personnel at (level/function; e.g., executive level, street-level, supervisory function, etc.) who implement (activity/behavior, which varies by level/function; e.g., number of meetings and content of meetings called by senior career managers, etc.) policies (specify policy area/type; e.g., hiring policies, procurement policies, etc.) decided by top political leadership (need operational definitions of “decided” and “top political leadership”). Actually, it’s better—and there’s not space here to do it—to divide this indicator into a number of indicators about policy/decision making and implementation processes and the type/level of personnel involved at various steps in these processes.
7. “Public sector employees strive to fulfill the ideology of the party/parties in government.” becomes: Percent of public sector personnel at (level/function; e.g., executive level, street-level, supervisory function, etc.) who, by (action/behavior, which varies by level/function; e.g., content of middle managers’ meetings, etc.) implement stated policy preferences of the party/parties in government.

When we translate survey research questions into statements that can be checked or verified by proper procedures and when we operationalize our words and concepts–such as “employed”, “personnel”, “implement”, etc.–we make our investigations more scientific, more empirical, and,thus, more useful to governments and citizens.

 

About georgebeam

George Beam is an educator and author. The perspectives that inform his interpretations of the topics of this blog–-as well as his other writings and university courses -–are system analysis, behaviorism, and Internet effects. Specific interests include quality management, methodology, and politics. He is Associate Professor Emeritus, Department of Public Administration; Affiliated Faculty, Department of Political Science; and, previously, Head, Department of Public Administration, University of Illinois at Chicago
This entry was posted in content analysis, Survey Research and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s