Wednesday, January 09, 2008

Political Pundits Worse Than Flipping a Coin

Via LeisureGuy, Jonah Lehrer:

Needless to say, the political pundits were hilariously wrong about the New Hampshire primary. I won't hypothesize about what actually happened, other than to say that I think many voters here wanted a longer primary. They went meta on the race and decided that they didn't want to coronate Obama in the beginning of January. This says less about Obama and Clinton and more about the over-hyped press coverage and shortened primary schedule. I voted for Obama, but I'm looking forward to a drawn out race for the Democratic nomination. This whole democracy thing is pretty entertaining.

But back to the failures of the political pundits. Nobody should be surprised. In 1984, the Berkeley psychologist Philip Tetlock began an epic research project: He picked two hundred and eighty-four people who made their living "commenting or offering advice on political and economic trends" - they were professional pundits - and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the experts were asked to rate the probability of several different possible outcomes. Tetlock then interrogated the experts about their thought process, so that he could better understand how they made up their mind. By the end of the study, Tetlock had quantified 82,361 different predictions.

After Tetlock tallied up the data, the predictive failures of most experts became painfully obvious. When asked to forecast the probability of a specific event happening, pundits tended to perform worse than random chance. A dart throwing chimp would have beaten the majority of well-informed experts. Tetlock also found that academic specialists - say, an expert in Middle Eastern affairs or a specialist on the New Hampshire primary - weren't any better than the-man-on-the-street at predicting the future. "We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly," Tetlock writes in Expert Political Judgment. "There is no reason for supposing that contributors to top journals--distinguished political scientists, area study specialists, economists, and so on--are any better than journalists or attentive readers of The New York Times in 'reading' emerging situations." Furthermore, the most famous experts in Tetlock's study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.

So think about that the next time you watch those talking heads on CNN.


Now I was as wrong about New Hampshire as all the professional pundits, but then again, I'm not being paid hundreds of thousands of dollars for my predictions and analysis, either.

The media do not have a liberal bias. They have a stupid bias.

8 comments:

Lawyer-Wearing-Yarmulka said...

The media do not have a liberal bias. They have a stupid bias.

Meh. Same thing.

(Sorry, I couldn't resist)

Yehudi Hilchati said...

Keep in mind that the pundits were basing their opinions on the polls. The polls were wrong on this one. The pundits blew hot air, as usual, but they were basing it on actual numbers gathered.

Yehudi Hilchati said...

I found an article about the research: http://www.project-syndicate.org/commentary/tetlock1

Comrade Kevin said...

The saying I'm reminded of a saying in this situation. It applies to gatherings of people in local settings primarily, but it has a larger context as well.

An expert is a person from more than fifty miles away who carries a briefcase.

Gorilla Bananas said...

I always knew Hillary would win. She's the kind of female I'd like in my band - her eyes would make a snake shiver.

zero said...

Hey Jewish Atheist, can you make a post about what you feel an atheist's attitude towards Israel should be. After all, Zionism stresses the return to historical homeland, and there is a whiff of Messianic ideology in that.

zero ++ said...

Or at least, the idea that Jews have traditionally hopped to return to the Holy Land of their fathers because that was allegedly what God promised.
The connection to the land was always religious. Hitchenson mentions this idea between the lines.

zero -- said...

Otherwise, why not Uganda or the US.