Friday, July 23, 2021

We show that an accurate prediction about such an extreme event, e.g., a big hit, may in fact be an indication of poor rather than good forecasting ability

From Predicting the Next Big Thing: Success as a Signal of Poor Judgment by Jerker Denrell and Christina Fang.  From the Abstract.

Successfully predicting that something will become a big hit seems impressive. Managers and entrepreneurs who have made successful predictions and have invested money on this basis are promoted, become rich, and may end up on the cover of business magazines. In this paper, we show that an accurate prediction about such an extreme event, e.g., a big hit, may in fact be an indication of poor rather than good forecasting ability. We first demonstrate how this conclusion can be derived from a formal model of forecasting. We then illustrate that the basic result is consistent with data from two lab experiments as well as field data on professional forecasts from the Wall Street Journal Survey of Economic Forecasts.

Reminds me of the tiny but interesting community of readers who spring into action whenever a consequential and unexpected thing happens.  They highlight some book, usually fiction, which closely mirrored the course of events which were unexpected, published years before the event.  Sometimes the parallels are eery.

One example is The Wreck of the Titan: Or, Futility a novel written by Morgan Robertson and published as Futility in 1898 (the Titanic sank in 1912).  From Wikipedia:

Although the novel was written before RMS Titanic was even conceptualized, there are some uncanny similarities between the fictional and real-life versions. Like Titanic, the fictional ship sank in April in the North Atlantic, and there were not enough lifeboats for all the passengers. There are also similarities in size (800 ft [244 m] long for Titan versus 882 ft 9 in [269 m] long for the Titanic), speed, and life-saving equipment. After the Titanic's sinking, some people credited Robertson with precognition and clairvoyance, which he denied. 

I think it is Tetlock, or perhaps Taleb, who has written about the contrasting pressures competing between accurate forecasting and forecasting which attracts attention.  Tetlock points out that in the media, the experts and pundits who receive the most attention are overwhelmingly incorrect in their forecasts whereas those who actually do a good job of forecasting, he calls them superforecasters, rarely receive attention.

This follows the model of hedgehogs and foxes ("a fox knows many things, but a hedgehog knows one big thing.")  

Partly this is because pundits and experts tend to be hedgehogs.  They have one narrow trick which they use all the time and is occasionally useful.  They come from a narrow field and they are indeed expert in that field but they mostly prognosticate about matters outside the field or about merely analogous events to their field of expertise.  Which is why they end up so often being wrong.  They also tend to be hermetically sealed in their knowledge bubble, not discovering much beyond what they already think they know.  

Among the more obvious examples is Paul Krugman, a once upon a time esteemed academic economist and winner of the Nobel Prize in Economics.  Then he transitioned into the, presumably, much more profitable role of pundit and mainstream media essayist where his track record for accurate forecasting, despite his credentials as an economist, has been disastrously bad.  

The most glaring example was perhaps on November 9th in his New York Times column, What Happened on Election Day.

It really does now look like President Donald J. Trump, and markets are plunging. When might we expect them to recover?

Frankly, I find it hard to care much, even though this is my specialty. The disaster for America and the world has so many aspects that the economic ramifications are way down my list of things to fear.

Still, I guess people want an answer: If the question is when markets will recover, a first-pass answer is never.

This is a forecast about an economy which then saw one of the historically best performances over the next four years, (till Covid-19), for both Main Street and for Wall Street.  Markets boomed, interests were low, employment high, job's plentiful, wages rising, etc.  

One of our most credentialed economic experts was disastrously wrong.  Which all hedgehog pundits often are.  They get attention by making strong claims on weak evidence, frequently bad evidence.  They get their click and attention but do not deliver useful forecasts.

Krugman's error was exacerbated by his explanation for the basis of his forecast.

Under any circumstances, putting an irresponsible, ignorant man who takes his advice from all the wrong people in charge of the nation with the world’s most important economy would be very bad news. What makes it especially bad right now, however, is the fundamentally fragile state much of the world is still in, eight years after the great financial crisis.

Clearly, his forecast was in part based on an emotional antipathy widely shared in the mainstream media and the Mandarin Class, later dubbed Trump Derangement Syndrome (TDS).  It is interesting to note that the economic fragility following the great financial crisis were dominated by Krugman's economic policies.  If one judged only outcomes, it was clear that Trump knew much more about creating a strong economy that benefitted everyone than did Nobel-prize winning Krugman.

In contrast to hedgehog pundits, Superforecasters tend to be classic foxes.  They are bright but not necessarily pinnacle bright.  While they have some specialization, they tend to read broadly and have a wide experience base.  They are curious for answers, not imposing answers.  

One of Tetlock's findings from the Good Judgment Project was that cognitive and personality traits were more important than specialised knowledge when it came to predicting the outcome of various world events typically more accurately than intelligence agencies. In particular, a 2015 study found that key predictors of forecasting accuracy were "cognitive ability [IQ], political knowledge, and open-mindedness". Superforecasters "were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness". In the Good Judgment Project, the superforecasters "scored higher on both intelligence and political knowledge than the already well-above-average group of forecasters" who were taking part in the tournament.

They are more interested in making accurate forecasts than in proving a point.


UPDATE:  How could I have forgotten some of Krugman's other clunkers.  From Why most economists' predictions are wrong by Paul Krugman.  From June 1998.  He followed the column with some forecasts in which he had great confidence.  Forecasts which then validated the headline of his essay.

Inflation will be back. Wages are rising at almost 5 percent annually, and the underlying growth of productivity is probably only 1.5 percent or less. Sooner or later, companies will have to start raising prices. In 1999 inflation will probably be more than 3 percent; with only moderate bad luck--say, a drop in the dollar--it could easily top 4 percent. Sell bonds! 

[SCORE - WRONG:  1997's inflation rate was 2.3%.  The average over the following 22 years was 2.1%.  1999's inflation was 2.2%]

Within two or three years, the current mood of American triumphalism--our belief that we have pulled economically and technologically ahead of the rest of the world--will evaporate. All it will take is a few technological setbacks or a mild recession here while Europe or Japan recovers a bit.  

[SCORE - WRONG: We had the Asian Financial Crisis of 1997, the Y2K scare, the tech recession of 2000, and the Home Mortgage bust of 2008.  But over that time frame, the American economy did consistently better than just about every other developed nation economy.]

The growth of the Internet will slow drastically, as the flaw in "Metcalfe's law"--which states that the number of potential connections in a network is proportional to the square of the number of participants--becomes apparent: most people have nothing to say to each other! By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's.  

[SCORE - WRONG: Spectacularly wrong.]

As the rate of technological change in computing slows, the number of jobs for IT specialists will decelerate, then actually turn down; ten years from now, the phrase information economy will sound silly. 

[SCORE - WRONG: The jobs with the highest incomes and most reliable demand and hottest demand over the subsequent 22 years have been in tech.]

Sometime in the next 20 years, maybe sooner, there will be another '70s-style raw-material crunch: a disruption of oil supplies, a sharp run-up in agricultural prices, or both. And suddenly people will remember that we are still living in the material world and that natural resources matter.

[SCORE - WRONG: Spectacularly wrong and then Almost right.  From 1998 to 2020, cost of living, and more particularly cost of materials have all fallen.  For a long stretch, many economies suffered deflation.  This is a particularly egregious forecasting failure given that the lesson of the Simon-Ehrlich wager was resolved in 1990 to the effect that in a modern competitive economy, the cost of materials in real terms will fall.  Almost right in the sense that the Biden administration (23 years after the forecast) appears, for the first time since the Carter years, well on the way to delivering high inflation and low economic growth, i.e. a return to stagflation.]

 

No comments:

Post a Comment