Wednesday, September 13, 2017

The Hurricane Irma Forecast: Triumph or Disappointment?

The quality of the National Weather Service forecast model projections for Hurricane Irma have received a lot of comments in the mainstream and social media.  Some have been positive, but others have been critical, suggesting that the NWS models have performed poorly, lagging behind the well-known European Center model.



Others have criticized the changing hurricane track, particularly the uncertainty over which side of Florida that hurricane would traverse.


Other media outlets noted apparent errors in the storm surge forecasts.

So how good were the American and other weather forecast models when it comes to Irma?
Have we made progress? 
Are there still problem with U.S. forecast models, something that became well-known during Hurricane Sandy in 2012? 

This blog will take on these questions.

My bottom line:  the Irma forecast was a triumph for the weather prediction community, but it also revealed continued problems with U.S. numerical weather prediction and our ability to communicate the uncertainty in model forecasts.

The Triumph

In many ways, the extended forecast of Hurricane Irma was an extraordinary triumph for weather prediction technology, with major global modeling systems (e.g., US GFS, European Center, Canadian, UKMET) suggesting a major threat to Florida a week or more out.   Even a decade ago, we could not have done this well.


Let us begin with a review of Irma's track (see below with legend).  Starting offshore of Africa as weak tropical disturbance, it headed westward, revving into a hurricane east of the Cape Verde Island, and by the time it approached Puerto Rico, Irma had exploded into a category 5 storm.   Subsequently, it moved WNW until it paralleled the northern Cuban coast before taking a sharp right turn that sent it across the Florida Keys and then northward over the western side of the Peninsula.  Irma made landfall on Florida on Sept. 10th.


Below are the ensemble track forecasts for Irma from the U.S. GEFS system (21 members or individual forecasts) and the European Center (ECMWF, 51 members) for the ten-day forecasts initialized on August 30th at 1200 UTC.  We use ensemble forecasts to get an idea of forecast uncertainties and to produce probabilities.

Starting with the European Center ensemble, although uncertainty increases in time, most of the ensemble members are taking a strong storm towards Florida.  A big warning MORE THAN TEN DAYS AHEAD of U.S. landfall.  Amazing.


The smaller U.S. ensemble (GEFS) initialized at the same time is also bringing Irma towards the U.S., but has a greater tendency to bring to storm up the Atlantic coast.


Now, let's examine the ensemble predictions for the two systems for the forecasts initialized two days before landfall (Sept 8th at 0000 UTC).  I will use the wonderful graphics produced by Professor Brian Tang of University of Albany (as an aside I just finished a wonderful visit there, meeting with Brian and his colleagues/students).  The individual ensemble are shown by the thin white lines and probabilities based on these tracks are shown by the shading.  The official (National Hurricane Center) forecast is shown by the black line and high-resolution version of the modeling system by the red-dashed line.

Starting with the European Center forecast, the model predicted the sharp right turn and the high probability for the storm to pass along the western half of Florida. Quite good.


The U.S. GEFS forecast also had a right turn, but it was taking the storm more along the eastern side of Florida, which was not correct.


Although the European Center solution was clearly superior, both U.S. and EC forecasts are very good....showing the threat to the U.S. more than a week ahead of time and predicting a sharp right turn days before. Other major modeling systems, such as the United Kingdom and Canadian models, did the same thing.  Predicting the exact location of the right turn days ahead is simply beyond the science at this time and may always be, but the models were all excellent in predicting that such a turn would occur in the vicinity of Florida.

A triumph for the technology of numerical weather prediction, with substantial credit going to those who have built the complex observing and modeling systems that made this possible.

Whispering warnings

During ancient Roman triumphs, a slave would stand behind the victorious general whispering in his ear "remember you are mortal" and I will act in this role now, at least for the American conquerors.

The U.S. global model was clearly inferior to the European Center model for this hurricane, as it was for Hurricane Sandy, Hurricane Harvey last month, and for many others.

To show this, here are the forecast errors for the hurricane locations (track errors)for the high-resolution forecasts of various global models and some of the U.S. hurricane models for 12 to 120 hours (the graphic produced by Brian Tang).  Track errors increase with time, as would be expected.  The U.S. global model (GFS, shown as AVNO--dark red) track errors are MUCH larger than the European Center (orange, ECWF), particularly for the longer forecasts (370 km error for the U.S. and 185 km error for the EC at 120 hours).  For many hours, the U.S. track error is TWICE the EC error.

Just as concerning, the high-resolution U.S. hurricane models had track errors that were substantially worse than those of ECMWF, including the newest U.S. hurricane model (HMON, light green) and the model that was developed (at a cost of tens of millions of dollars) over the past five years, HWRF (aquamarine).

Hurricane Harvey?   Simliar story,  with HMON going wacky at some hours.

Many of the HMON forecasts were completely out to lunch, producing unphysical results.  For example, here are the pressure forecasts for HMON (green), HWRF (purple),  and observed (black).  HMON took the storm down to a completely unrealistic 850 hPa central pressure (observed was around 930 hPa).  With crazy pressures and often large track errors, HMON clearly has very serious problems and should not be shown publicly.  Why it is even necessary is another major question.

 The general superiority of the European Center model is also suggested by other statistics, including the 48h track errors for tropical storms and hurricanes produced by the National Hurricane Center for the last several decades.   Clearly, there has been great progress since 1984, with track errors going from around 200 km to less than 100 km.   That is the triumph stuff.  But this sample of many storms shows that the European Center is consistentlythe best (light blue dots).  The EC would be even more dominant at longer projections.


So, if was whispering in the ears of an American weather general, I was note the following:

  • The ability of the U.S. to forecast hurricanes has clearly improved. Congratulations.
  • Five years after Hurricane Sandy, the U.S. has not caught up to the European Center, with both improving at roughly the same rate.   We need to do better.
  • The European Center does a better job at assimilating a wide variety of observations and their model has better physics (e.g., descriptions of moist processes) than the U.S. models--we need to up our game.  The proposed new U.S. modeling system (FV-3) is not going to fix these areas.
  • The U.S. has spent large amounts of money on hurricane models (e.g., HWRF and now HMON), but in many ways they are inferior to a coarser global model (EC), particularly for track forecasts.  
  • A clearly deficient hurricane model has recently been developed (for reasons that are not clear)...HMON.  It is not ready for prime time, so why show it?
  • There was a lot of confusion about which side of Florida would be hit by Irma.  Much of this confusion can be traced to inadequate communication by the National Weather Service and the media, with substantial misunderstanding of ensemble prediction by the lay community.  We need to do much better in this domain (more in a future blog).
  • National Weather Service verification of model quality and hurricane forecast skill is very poor.  Why do we have to depend on an innovative professor (Brian Tang), and unofficial web sites (e.g., weathernerds) to supply such information?
A triumph for sure but many problems remain for U.S. hurricane and numerical weather prediction.



13 comments:

windlover said...

I followed Irma quite closely because my father-in-law and son of a good friend are both in Florida. Yes, the track kept changing. But every time they updated they (the weather channel) made it very clear that the path could change quickly so the entire state should prepare. That they couldn't give the exact track and that is the reason for the cone getting wider the further out the forecast was. They made it very clear that the entire state of Florida and other coastal states would be affected because of the size and strength of this storm. When the storm was 24 hours out and it was clear it was going to go up the West side of the state I was shaking my head reading about the West coast residents who hadn't boarded up Windows or made other preparations because "they said it was going to hit the east side". Obviously they either weren't paying attention or didn't watch the reports closely/often enough. I thought they kept us extremely well informed. People need to also keep in mind that this is mother nature we are dealing with. She does what SHE wants...not what the models tell us she will do.

JeffB said...

The problem is not the predictions. The problem is what the media does with predictions. Often blown way out of proportion. Or just another drum beat for AGW, even as there may be no immediate connection. And so the average person grows weary in the fog of media hyperbole.

Eric Blair said...

Having read a discussion on the continuing disparity between the US and Euro Climate modeling systems, one senior European official said (off the record), that at this point it wasn't a question of technological prowess, but one of staffing. Apparently the Euros still employ more people overall, so his point was that you get what you pay for in this instance.

K.R. Burgess said...




**************** TRIUMPH **************

Another GREAT blog,by Dr. Cliff Mass !!!

David Cuthbert said...

Perhaps a silly question: Why can't the US models incorporate parts (if not the source, at least the published algorithms and ideas) of the ECMWF? Or does the ECMWF even publish this? (Looks like they rely on paid forecast subscriptions, so it might not be in their interest.)

I see that the NOAA models are available for download (http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/), which isn't surprising (many federal agencies are required to make their work open).

From my days in the electronic CAD world, once someone published an algorithm that was demonstrably superior, everyone was rushing to get it reimplemented in their software (either evading patents or hoping others wouldn't notice).

Bruce Kay said...

JeffB - The average consumer does not grow weary, they lap it up like ice cream, which why the media hypes it up!

Lets get real here. Like everything we follow, we expect a good competition where there are winners and losers, certain models getting a pat on the back while the others get sent off to the dog house. That is how we follow everything from celebrity to politics. A football game, in other words.

How about this? By any measure of predicting the barely predictable, even the lousy models can only be described as highly valuable and in the money from a risk planning stand point. Someone already mentioned a "cone of error" as a zone where we should expect the unexpected.... and plan for it. It happened. The models were spectacularly successful on everything from tracking to destructive potential.

The fact that the east coast missed a direct hit is no error in modelling - the modelling suggested exactly that possibility and only a fart in the wind made it go elsewhere. These events are no football game - they are risk problems of the highest order and they should be treated so. The ensemble is a collaboration of effort, not a freaking competition.

John K. said...

When it comes to the "media", try to remember the old adage.. "never wrestle with a pig - you will get dirty and the pig likes it".

Why is there so much concern about what the "media" reports? Do you really expect them, with their knowledge and understanding of these subjects, to get this right?

Just turn them off and go about your business.. click.

Elston Hill said...

You are in a tough profession. As an amateur photographer, I am always checking forecasts before traveling to locations and am constantly amazed at the accuracy of the predictions. I depend on the forecasts to get the pictures I want. I love the Weather Underground. And then I hear people make snide remarks about weather predictions. That does not jive with my experience. I do take the time to check hourly forecasts and keep monitoring those forecasts on my smart phone. Your profession makes my job much easier. Thanks.

John Marshall said...

Before modern numerical forecasting, Irma would have been predicted to just keep going west. That's what it had been doing since it was on the other side of the Atlantic. It wouldn't be until it hit the Keys that someone would have realized that it had made a hard turn, and then it would be too late for Florida. I think the 1935 Labor Day hurricane (Cat 5 that hit the Keys) was perhaps a case in point. Forecasters knew it was out there, but didn't forecast a change in track properly or its intensity. That was, of course, in the days before computers.

But the models predicted a turn for Irma, a sharp one, and it was neat to watch Irma hang a right pretty much exactly as predicted. So they were off a few degrees on the final course. That's nit compared to predicting that huge turn up into Florida almost a week ahead of time.

Very impressive.

Bruce Kay said...

I find this search for perfection (in a domain where perfection is unattainable) to be fascinating and a bit disturbing. On one side it makes sense - how can we possibly lose with ever greater accuracy? We wouldn't - if the problem was a perfect equation where ever factor is known.

It is not. Any weather is a risk problem and risk always involves unknowns. It has been found in modern aviation that the ever increasing reliability and reliance on automation in everything from navigation to mechanics comes at a price. The price is eroding expertise from the human pilot. Skill is not so much lost as never attained quite as much as before. Skill requires practice with feedback. Take that away from the pilot and he/she does not attain skill.

Imagine if we became so in love with the Euro model that works so wonderful and seems near perfect to the point that we rely on it confidently. Perhaps evacuating the whole west side with confidence and forgetting all about Miami.... until it was clear that the Euro model was finally wrong for once. I really doubt the meteorologists would fall for that but the decision makers - the risk managing pilots - could as their only feedbacks iare a model that always works.

Nobody follows a compass bearing in a whiteout without a clear understanding of where error might lead you - if you're smart anyway. These days there are plenty of stories of the not quite so smart following their perfect GPS straight off of cliffs. They are trained to expect no error.

BAMCIS said...

It would be fantastic to see more funding for meteorology and other sciences. Alas, we live in a country that celebrates ignorance, bullies intelligence and elects people like Donald Trump. Our current administration might sooner shut the NWS down, outsource forecasting to Europe with the caveat to omit anything that might disagree with prophecy/ideology or faith. Certainly no data shall see the light of day that suggests our climate is dynamic...and that we Humans are helping things along a bit. That would be bad for business!

The Meteorology disciplines did a fine job and saved countless lives these past few weeks, but don't put that past the powers that be. The politicians will be sure to down play any victories science achieved, as well as learn nothing in retrospect. Insurance will build it all back just how it was, and everyone can just forget it ever happened and live happily ever after.....until the next angry storm named after someone's friendly elderly relative comes knocking. That OK, though. Think of all those construction jobs Donald Trump can take credit for. The cure is more lucrative than the prevention.

Miami will still continue to build like tomorrow never comes. So will Houston.

Shelley said...

Cliff-can you recommend a site for the general public to look for longer range weather predictions? I work at a native plant nursery in Bellingham and it would be helpful to have an idea of when we might get our first frost, and a general idea of how cold (freezes, snow) we might be during the winter. It would help us anticipate conditions for digging bare root plants during the winter. I don't know if there is a site for this, just wondering (and hoping). Last winter we were frozen most of December and January, so we couldn't dig plants. We weren't expecting that, and it hadn't happened before-such a long time with our ground frozen. Thanks.

Joseph Ratliff said...

Hi Shelley,

Sorry so late, but try here:

http://www.cpc.ncep.noaa.gov

Hover your mouse over the words "Temperature" and "Precipitation" in each of the blue blocks.

You'll see the image change with each hover. If you click, you'll bring up a close up of the image forecast.