Humility is called for but there is little market for humility when it comes to money and power.

If accurate, this is interesting and worthwhile information. From US forecast models have been pretty terrible during Hurricane Irma by Eric Berger.

Berger is using Irma as a case study for comparison among nearly a dozen hurricane forecasting models out of the US and Europe.

We have written a fair amount at Ars recently about the superiority of the European forecast model, suggesting to readers that they focus on the ensemble runs of this system to get a good handle on track forecasts for Hurricane Irma. Then we checked out some of the preliminary data on model performance during this major hurricane, and it was truly eye-opening.

Brian Tang, an atmospheric scientist at the University of Albany, tabulates data on "mean absolute error" for the location of a storm's center at a given time and where it was forecast to be at that time. Hurricane Irma has been a thing for about a week now, so we have started to get a decent sample size—at least 10 model runs—to assess performance.

[snip]

Forecast models typically show their skill with three-, four-, and five-day forecasts. For simplicity's sake, we will focus on 120-hour forecasts. At this lead time, the average error of the European model with respect to Irma has been about 175km in its position forecast. The next best forecast is from the hurricane center, which is slightly more than 300km. An automated model, then, has so far beaten human forecasters at the National Hurricane Center (looking at all of this model data) by a wide margin. That's pretty astounding.

What is particularly embarrassing for NOAA, however, is the comparison between the European model and the various US forecast modeling efforts. The average 120-hour error of the GFS model is about 475km. The operational, hurricane-specific model, HWRF, does better, with an average error of 325km. But the experimental HMON model does terribly, at nearly 550km of error. A similar disparity in quality goes all the way down to 24-hour forecasts.

[snip]

So what's the deal here? The overall performance of the National Weather Service's GFS model has lagged for years behind the European forecast system, which is backed up by superior resources and computing power. Finally, this year, the GFS was upgraded. However, even before those upgrades went into effect, hurricane forecasters were raising concerns about the new GFS.

Shortly before the beginning of the 2017 Atlantic hurricane season, in fact, forecasters at the National Hurricane Center in Miami pushed back against the upgrade. They had noted degraded performance during internal tests of the GFS model on Atlantic tropical cyclones. The track forecasts were about 10 percent worse with the newer version of the model than the older one.

In a presentation posted on the National Weather Service website, first reported by Mashable, the hurricane center officials said, "The loss of short- to medium-range [tropical cyclone] track and intensity forecast skill for the Atlantic basin in the proposed 2017 GFS is unacceptable to the National Hurricane Center." Ultimately, the upgrade was initiated anyway.

There are all sorts of caveats to be attached to this report. I have no contextual knowledge of Berger as a reporter nor do I have domain knowledge about hurricane forecasting. But I am reasonably knowledgeable about forecasting as a discipline (economic forecasting, business forecasting, policy forecasting, etc.)

Complex systems are notoriously difficult to predict with precision and are usually especially sensitive to time horizons - the further out you are forecasting, the more likely it is that unknown aspects of the system will affect the forecast and the more likely it is that unrelated exogenous shocks will have a material impact.

In forecasting you are interested in, among other things, the precision and the accuracy of the forecast. Very broadly, precision is how tightly clustered are the various forecasts and accuracy is how close the cluster is to the actual result. With this type of interest, you have four possible outcomes.

Click to enlarge.

According to Berger's information, the very best five-day forecast (from Europe) is about 110 miles off base. The standard American forecast is nearly three hundred miles off. Given the wide range of forecasts among the eleven reported, it seems like it would be fair to say that the science of hurricane forecasting, while dramatically better than it was even twenty-five years ago, is still low precision and low accuracy.

A hundred to three hundred mile variance is the difference between billions of dollars of damage, dozens or hundreds of lives lost, and even more billions of dollars of lost productivity if the hurricane travels over a densely populated urban area and millions of dollars of damage and maybe few or no lost lives if it goes over a rural low population area.

My point is not to denigrate the science of hurricane forecasting. It is a tremendously complex system and forecasts have been getting better over time.

My point is that we have immediate and very tactical interests in getting hurricane forecasting to be more precise and accurate, we have spent more than a hundred years attempting to improve forecasts, and yet the complexity of the system is such that we are still nowhere near the precision and accuracy that would be very useful.

With the current track of Irma compared to what we thought five days ago, hundreds of thousands of people, perhaps in excess of a million, from eastern Florida and southeastern Georgia are on the road with lost money, lost productivity, and health and danger exposures that in hindsight we seem not to have needed to evacuate. Hurricane Irma is tracking further and further west than initially forecast. Bad for Tallahassee and Mobile but great for the much more densely populated Miami, Jacksonville, Savanah, Macon, and Atlanta as originally forecast.

Despite the huge and immediate and very human need for better hurricane forecasts, and despite years of heavily funded effort, we are nowhere near the level of understanding we need to achieve.

Anthropogenic Global Warming is based on model forecasts of systems which are enormously more complex than those associated with hurricanes; for which data is far more partial and fragmentary; for which there are far more confounds, about fields of science which are still in their nascent stages; about which are hypotheses and theories are still nascent; and for which even the shorter term decadal forecasts have been dramatically wrong.

What cause is their for us to have greater confidence in AGW forecasting than we have in the demonstrated low precision and low accuracy of hurricane forecasting? There isn't a good cause.

We ought to reduce emissions to the extent that can be economically justified based on short (yearly) or medium term (decadal) time frames. Anything beyond that involves faith based forecasts of dubious precision or accuracy.

The absence of precision and accuracy of hurricane forecasting is an example that serves to remind us that forecasts of centennial and millennial frames need to be treated with due precaution.

The prescriptions associated with AGW are so consequential (diversions of billions of dollars from productive use to uncertain future benefit) that they should only be undertaken with a great deal of confidence. In contrast, the substance of the debate is still primarily that of a faith-based ideology. The public in most OECD countries are highly skeptical abut the underlying science and track record of forecasting but the vested elite are completely wedded to idea that the science is settled even though they know that there is nothing like a sure thing and even though they know that the science is never settled.

Humility is called for but there is little market for humility when it comes to money and power.