Political Pundits Worse Than Flipping a Coin

Via LeisureGuy, Jonah Lehrer:

Needless to say, the political pundits were hilariously wrong about the New Hampshire primary. I won't hypothesize about what actually happened, other than to say that I think many voters here wanted a longer primary. They went meta on the race and decided that they didn't want to coronate Obama in the beginning of January. This says less about Obama and Clinton and more about the over-hyped press coverage and shortened primary schedule. I voted for Obama, but I'm looking forward to a drawn out race for the Democratic nomination. This whole democracy thing is pretty entertaining.

But back to the failures of the political pundits. Nobody should be surprised. In 1984, the Berkeley psychologist Philip Tetlock began an epic research project: He picked two hundred and eighty-four people who made their living "commenting or offering advice on political and economic trends" - they were professional pundits - and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the experts were asked to rate the probability of several different possible outcomes. Tetlock then interrogated the experts about their thought process, so that he could better understand how they made up their mind. By the end of the study, Tetlock had quantified 82,361 different predictions.

After Tetlock tallied up the data, the predictive failures of most experts became painfully obvious. When asked to forecast the probability of a specific event happening, pundits tended to perform worse than random chance. A dart throwing chimp would have beaten the majority of well-informed experts. Tetlock also found that academic specialists - say, an expert in Middle Eastern affairs or a specialist on the New Hampshire primary - weren't any better than the-man-on-the-street at predicting the future. "We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly," Tetlock writes in Expert Political Judgment. "There is no reason for supposing that contributors to top journals--distinguished political scientists, area study specialists, economists, and so on--are any better than journalists or attentive readers of The New York Times in 'reading' emerging situations." Furthermore, the most famous experts in Tetlock's study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.

So think about that the next time you watch those talking heads on CNN.

Now I was as wrong about New Hampshire as all the professional pundits, but then again, I'm not being paid hundreds of thousands of dollars for my predictions and analysis, either.

The media do not have a liberal bias. They have a stupid bias.