EDITOR'S CONTRIBUTON: Yet Another Danger of Polls

By Andrew Collins

Some politicians like to avoid risk by heeding the advice of polls. But evidence keeps suggesting that the real risk may lie in following them too closely.

Take a look at this result in “The Arkansas Poll,” administered in 2005 by political scientists Janine A. Parry and Bill Schreckhise at the University of Arkansas. The question is:

Do you approve or disapprove of a law that would prohibit Arkansas textbooks from defining marriage as anything other than a relationship between one man and one woman?

(In case you were unable or unwilling to follow that, “approve” is basically pro-gay marriage, and “disapprove” is basically anti-gay marriage.)

“The Arkansas Poll” broke down the results by “all respondents,” “evangelicals/‘born agains,’” “weekly [or more] churchgoers,” “[people who] know a gay or lesbian person,” and “[people who have a] somewhat/very unfavorable view of gays/lesbians.” From common sense, you might expect evangelicals to feel less favorably about teaching about alternative marriage, and you might expect people who know a gay or lesbian person to feel more favorably.

You would be wrong.

According to the poll, 81 percent of evangelicals/“born agains” take the essentially pro-gay view, along with 83 percent of people who have a somewhat/very unfavorable view of gays/lesbians. Specifically, the vast majority of evangelicals and anti-gays think Arkansas textbook writers should be free to define marriage as something other than a relationship between one man and one woman. This is far greater than the 70 percent of all respondents who feel this way.

This is a preposterous result. And it gets worse. Among those people who know a gay or lesbian person, a mere 68 percent approve of the law. According to the poll, people who personally know gays are less likely than people in any other group to support the teaching of alternative marriage.

Something is obviously amiss. The people of Arkansas are generally intelligent, thoughtful about gay issues, and honest, so that leaves two possibilities. Maybe evangelicals and anti-gay people actually favor gay marriage at unusually high rates, and people who know gays are more likely to oppose gay marriage. Or maybe the poll is misleading.

Let’s give the first explanation a fair shake, however unlikely it sounds. After all, if all we needed were our preconceptions, polls would be superfluous. So let us take a look at the answers voiced by various groups in the same University of Arkansas poll (broadly, a higher percentage signifies pro-gay attitudes):

• Forty-six percent of evangelicals, 38 percent of anti-gays, and 67 percent of people who know a gay or lesbian person approve of gays in the military.
• Sixty-five percent of evangelicals, 61 percent of anti-gays, and 76 percent of people who know a gay or lesbian person approve of gay adoption.
• Twenty-two percent of evangelicals, 14 percent of anti-gays, and 31 percent of people who know a gay or lesbian person would ban discrimination against gays in the workplace.
• Thirty-nine percent of evangelicals, 33 percent of anti-gays, and 51 percent of people who know a gay or lesbian person would permit gay foster parents.

Clearly, evangelicals and anti-gay people in Arkansas are consistently more opposed to gay rights than people who personally have gay acquaintances—just like our intuition tells us. So on the question of defining marriage in textbooks, the poll’s result is just plain wrong. The results on this particular question probably need to be reversed to reflect the true feelings of Arkansans. The poll question was flat misleading, to the point that it actually produced an opposite result.

(The poll’s lead researcher, Janine A. Parry, acknowledged in an email message that respondents did, indeed, have difficulty answering the question. “In fact,” she wrote, “we dropped those results from our academic analyses because people clearly misunderstood our meaning.”)

All this would be unfortunate but acceptable, of course, were the question not about a hot-button issue in a credible major university poll. A newspaper, politician, or researcher could selectively pull misleading data from the poll and present an entirely justified, entirely inaccurate result to the public or policymakers.

For example, anti-gay forces could use the misleading poll result to essentially say, “People who personally know gays oppose gay rights even more than the average Arkansan. You think gays are bad now—just wait until you meet one!” Or, pro-gay forces could say, “Even people who don’t like gays think it’s fine to teach kids about alternative marriages!” No one, on either side, wants such a ridiculous manipulation of facts. Using such invalid “results” could lead to the warping of voting patterns, legislative or referential agendas, or executive agendas for politicians who read too deeply into polls and not their own hearts.

There are two lessons to be drawn from this case. One, poll questions should be more clearly written so as to obtain more accurate answers. Two, politicians’ worship of polls represents a serious peril. Even when questions are phrased straightforwardly, results are not always what they appear to be. It is best for politicians to be true to their principles, express them clearly to the people, and let the polls take place where they belong: at the ballot box, where everyone has a voice in a vote.

Do you not approve of a ban on disallowing polls to unduly influence politicians from their course, with the expectation that they will heed them to the ignorance of the possibility that they may present misleading information?

My answer: Yes.