The Hurricane Irma Forecast: Triumph or Disappointment?

The quality of the National Weather Service forecast model projections for Hurricane Irma have received a lot of comments in the mainstream and social media.  Some have been positive, but others have been critical, suggesting that the NWS models have performed poorly, lagging behind the well-known European Center model.

Others have criticized the changing hurricane track, particularly the uncertainty over which side of Florida that hurricane would traverse.

Other media outlets noted apparent errors in the storm surge forecasts.

So how good were the American and other weather forecast models when it comes to Irma?
Have we made progress? 
Are there still problem with U.S. forecast models, something that became well-known during Hurricane Sandy in 2012? 

This blog will take on these questions.

My bottom line:  the Irma forecast was a triumph for the weather prediction community, but it also revealed continued problems with U.S. numerical weather prediction and our ability to communicate the uncertainty in model forecasts.

The Triumph

In many ways, the extended forecast of Hurricane Irma was an extraordinary triumph for weather prediction technology, with major global modeling systems (e.g., US GFS, European Center, Canadian, UKMET) suggesting a major threat to Florida a week or more out.   Even a decade ago, we could not have done this well.

Let us begin with a review of Irma's track (see below with legend).  Starting offshore of Africa as weak tropical disturbance, it headed westward, revving into a hurricane east of the Cape Verde Island, and by the time it approached Puerto Rico, Irma had exploded into a category 5 storm.   Subsequently, it moved WNW until it paralleled the northern Cuban coast before taking a sharp right turn that sent it across the Florida Keys and then northward over the western side of the Peninsula.  Irma made landfall on Florida on Sept. 10th.

Below are the ensemble track forecasts for Irma from the U.S. GEFS system (21 members or individual forecasts) and the European Center (ECMWF, 51 members) for the ten-day forecasts initialized on August 30th at 1200 UTC.  We use ensemble forecasts to get an idea of forecast uncertainties and to produce probabilities.

Starting with the European Center ensemble, although uncertainty increases in time, most of the ensemble members are taking a strong storm towards Florida.  A big warning MORE THAN TEN DAYS AHEAD of U.S. landfall.  Amazing.

The smaller U.S. ensemble (GEFS) initialized at the same time is also bringing Irma towards the U.S., but has a greater tendency to bring to storm up the Atlantic coast.

Now, let's examine the ensemble predictions for the two systems for the forecasts initialized two days before landfall (Sept 8th at 0000 UTC).  I will use the wonderful graphics produced by Professor Brian Tang of University of Albany (as an aside I just finished a wonderful visit there, meeting with Brian and his colleagues/students).  The individual ensemble are shown by the thin white lines and probabilities based on these tracks are shown by the shading.  The official (National Hurricane Center) forecast is shown by the black line and high-resolution version of the modeling system by the red-dashed line.

Starting with the European Center forecast, the model predicted the sharp right turn and the high probability for the storm to pass along the western half of Florida. Quite good.

The U.S. GEFS forecast also had a right turn, but it was taking the storm more along the eastern side of Florida, which was not correct.

Although the European Center solution was clearly superior, both U.S. and EC forecasts are very good....showing the threat to the U.S. more than a week ahead of time and predicting a sharp right turn days before. Other major modeling systems, such as the United Kingdom and Canadian models, did the same thing.  Predicting the exact location of the right turn days ahead is simply beyond the science at this time and may always be, but the models were all excellent in predicting that such a turn would occur in the vicinity of Florida.

A triumph for the technology of numerical weather prediction, with substantial credit going to those who have built the complex observing and modeling systems that made this possible.

Whispering warnings

During ancient Roman triumphs, a slave would stand behind the victorious general whispering in his ear "remember you are mortal" and I will act in this role now, at least for the American conquerors.

The U.S. global model was clearly inferior to the European Center model for this hurricane, as it was for Hurricane Sandy, Hurricane Harvey last month, and for many others.

To show this, here are the forecast errors for the hurricane locations (track errors)for the high-resolution forecasts of various global models and some of the U.S. hurricane models for 12 to 120 hours (the graphic produced by Brian Tang).  Track errors increase with time, as would be expected.  The U.S. global model (GFS, shown as AVNO--dark red) track errors are MUCH larger than the European Center (orange, ECWF), particularly for the longer forecasts (370 km error for the U.S. and 185 km error for the EC at 120 hours).  For many hours, the U.S. track error is TWICE the EC error.

Just as concerning, the high-resolution U.S. hurricane models had track errors that were substantially worse than those of ECMWF, including the newest U.S. hurricane model (HMON, light green) and the model that was developed (at a cost of tens of millions of dollars) over the past five years, HWRF (aquamarine).

Hurricane Harvey?   Simliar story,  with HMON going wacky at some hours.

Many of the HMON forecasts were completely out to lunch, producing unphysical results.  For example, here are the pressure forecasts for HMON (green), HWRF (purple),  and observed (black).  HMON took the storm down to a completely unrealistic 850 hPa central pressure (observed was around 930 hPa).  With crazy pressures and often large track errors, HMON clearly has very serious problems and should not be shown publicly.  Why it is even necessary is another major question.

 The general superiority of the European Center model is also suggested by other statistics, including the 48h track errors for tropical storms and hurricanes produced by the National Hurricane Center for the last several decades.   Clearly, there has been great progress since 1984, with track errors going from around 200 km to less than 100 km.   That is the triumph stuff.  But this sample of many storms shows that the European Center is consistentlythe best (light blue dots).  The EC would be even more dominant at longer projections.

So, if was whispering in the ears of an American weather general, I was note the following:

• The ability of the U.S. to forecast hurricanes has clearly improved. Congratulations.
• Five years after Hurricane Sandy, the U.S. has not caught up to the European Center, with both improving at roughly the same rate.   We need to do better.
• The European Center does a better job at assimilating a wide variety of observations and their model has better physics (e.g., descriptions of moist processes) than the U.S. models--we need to up our game.  The proposed new U.S. modeling system (FV-3) is not going to fix these areas.
• The U.S. has spent large amounts of money on hurricane models (e.g., HWRF and now HMON), but in many ways they are inferior to a coarser global model (EC), particularly for track forecasts.  
• A clearly deficient hurricane model has recently been developed (for reasons that are not clear)...HMON.  It is not ready for prime time, so why show it?
• There was a lot of confusion about which side of Florida would be hit by Irma.  Much of this confusion can be traced to inadequate communication by the National Weather Service and the media, with substantial misunderstanding of ensemble prediction by the lay community.  We need to do much better in this domain (more in a future blog).
• National Weather Service verification of model quality and hurricane forecast skill is very poor.  Why do we have to depend on an innovative professor (Brian Tang), and unofficial web sites (e.g., weathernerds) to supply such information?

A triumph for sure but many problems remain for U.S. hurricane and numerical weather prediction.