Innovation
Forecasting accuracy and a dart-throwing chimpanzee
By Michael Hennigan, Finfacts founder and editor
Dec 20, 2013 - 3:49 PM

Alan Greenspan finds a flaw in a 40-year old belief...

A study reported in Philip Tetlock’s 2005 book 'Expert Political Judgment: How Good Is It? How Can We Know?,' showed that the average expert was found to be only slightly more accurate than a dart-throwing chimpanzee. Many experts would have done better if they had made random guesses. Prof Tetlock writes in the Economist's 'The World in 2014' on who's good at forecasts. The Greek poet Archilochus had observed about 2,700 years ago: “The fox knows many things, but the hedgehog knows one big thing.” Isaiah Berlin in his famous 1953 essay “The Fox and the Hedgehog” contrasted hedgehogs that “relate everything to a single, central vision” with foxes who “pursue many ends connected…if at all, only in some de facto way.” This was an illustration of specialists and generalists at work.

“People are not as good at anticipating the future as they think they are,” reckons Tetlock who is a professor of psychology and management at the University of Pennsylvania, in his video on predictions (see beow). His assessment and piece he co-authored is in 'The World in 2014,' which explores political, economic and sporting possibilities of the year ahead.

The World in 2014: Who's good at forecasts?

Chapter 1: Expert Political Judgment: How Good Is It? How Can We Know? Philip E. Tetlock

More video: a conversation with Philip Tetlock   

There was     systematic overconfidence. Moreover, political analysts were disinclined to     change their minds when they get it wrong. When they made strong predictions that something was going to happen and it didn’t, they were inclined to argue something along the lines of, "Well, I predicted that the Soviet Union  would continue and it would have if the coup plotters against Gorbachev had  been more organized," or "I predicted the Canada would disintegrate or Nigeria would disintegrate and it's still, but it's just a matter of time before it disappears," or "I predicted that the Dow would be down 36,000 by the year 2000 and it's going to get there eventually, but it will just take a bit longer."   

So, we  found three basic things: many pundits were hard-pressed to do better than chance, were overconfident, and were reluctant to change their minds in  response to new evidence. That combination doesn't exactly make for a flattering portrait of the punditocracy.]

Louis Menand, a renowned author and Harvard academic, wrote in 'The New Yorker' in 2005: "Our system of expertise is completely inside out: it rewards bad judgments over good ones." In a review of Philip Tetlock 2005 book, he wrote, "that people who make prediction their business - - people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables - - are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons.

They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote."   

Prof Tetlock conducted     a 20+ year study of 284 professional forecasters. He asked them to predict the probability of     various occurrences both within and outside of their areas of expertise. Analysis of the 80,000+ forecasts found that experts are less accurate predictors than non-experts in their area of expertise.

The Economist's World in 2014: Philip Tetlock:

Tetlock's conclusion: when seeking accuracy of predictions, it is better to turn to   those like "Berlin's prototypical fox, those who know many little things, draw from an eclectic array of traditions, and accept ambiguity and  contradictions." Ideological reliance on a single perspective appears detrimental to one's ability to successfully navigate vague or poorly-defined situations (which are more prevalent today than ever before).

We wrote days before last November US elections that [President Obama is now better than a 4-in-5 favourite to win the Electoral College, according to the FiveThirtyEight forecast, made by Nate Silver the contributing to The New York Times. "His chances of winning it increased to 83.7% on Friday, his highest figure since the Denver debate and improved from 80.8% on Thursday." If on Wednesday Barack Obama is re-elected as America's 44th president (the 43rd occupant of the office), Nate Silver, a 34-year old statistician and economics graduate of the University of Chicago, will also be a winner while the traditional pundits will be among the losers.]

Erza Klein in the Washington Post wrote:   

If you had to distil the work of a political pundit down to a single question, you’d have to pick the perennial “who will win the election?” During election years, that’s the question at the base of most careers in punditry,  almost all cable news appearances, and most A1 news articles. Traditionally,  we’ve answered that question by drawing on some combination of experience, intuition, reporting and polls. Now Silver - - and Silver’s  imitators and     political scientists - - are taking that question away from us. It would be  shocking if the profession didn’t try and defend itself.]

Why Nate Silver's Gambling Streak Makes Me Trust Him More

Robert Shiller, Yale economist and Nobel laureate, says not one economist forecast the Great Depression:


© Copyright 2011 by Finfacts.com

Printer-friendly page from Finfacts Ireland Business News - Click for the News Main Page - A service of the Finfacts Ireland Business and Finance Portal