February 08, 2010

Humility and Forecast Busts

Weather Prediction is a Stern Master

Weather prediction is a business that promotes humility, and nothing brings one down to earth like a forecast failure--known as a "forecast bust" in the business. Major forecast failures are less frequent than even a decade ago...mainly due to a huge increase in satellite data and better computer models... but they still occur with discomforting frequency!

The most failure-prone forecasts are the most long-ranged ones...such as seasonal predictions done months in advance. Want a primo failure? In the fall of 2008, we were in a neutral ENSO (El Nino-Southern Oscillation) year and the long-range prediction was for a stormy, but normal winter. In December, there was an abrupt, unexpected turn to La Nina, with a change to cold and snow in the NW that took down our erstwhile mayor. Complete failure....embarrassing! (for both forecasters AND the mayor).

In contrast, this year everything went smoother. Moderate El Nino...with expectations in the NW for warmer and drier than normal conditions after January 1. Almost on schedule it turned warmer, but initially wetter. January was the warmest on record (much more than we expected). On the other hand the weather pattern of the last 3 weeks have been classic El Nino, with a split flow (jet stream going into AK and CA) and southern CA getting pummeled by strong storms. Passable grade?

Sometimes the most difficult forecasts are when there are weak systems floating around and the flow is modest. Like this weekend.

On Friday, both the NWS and I were going for essentially the same forecast. Mediocre Saturday as a weak system was coming in and a much better (dry, sun) on Sunday. The truth was almost reversed. Saturday became warm and sunny and Sunday was cloudy with rain in the morning.

Below are three computer forecasts for 24-h precipitation ending 4 PM on Sunday.

The first was made Friday morning. No rain except for the coast!

The second was made Saturday morning. Rain much more extensive.

The third was made Saturday night. Even more rain.

So you can understand how we went wrong on Friday. And you can see how the forecasts tended to get better as we got closer in time.

But my forecast failure was completely unnecessary (and yes so was the NWS's)

There is another way of forecasting...the right way...using a collection of forecasts..called ensembles. Run and use many forecasts all starting a bit differently or with slighly different model physics. If ALL of them are dry, then you go dry. If half of them are wet, 50% might be a good forecast. Etc.

Well, I sheepishly looked at the UW emsemble forecast system Sunday and sure enough some of the ensemble members on Friday were much wetter (see the ensemble member based on the Canadian model). And putting 8 members together the probability of precipitation for the two 12-h periods covering 5 PM on Saturday to 5 PM on Sunday (see graphics) had significant probablities of rain (at least 50%). So I SHOULD have forecast a significant chance of rain on Sunday. No more rushed forecasts for me. Until next time.

So the moral of the story...don't look at a single forecast model output. Look at many! Think probabilistically. Even though this technology is under development here at the UW, old habits are slow to die!




11 comments:

  1. you're an angel cliff, no need to worry. we had a glorious day saturday west of olympia, and a nice and warm if cloudy day sunday. no rain either day.

    thanks for all of the great info, you easily have the most useful local blog in the northwest.

    ReplyDelete
  2. Cliff. Having lived in the NW for 44 years, I have come to expect rain and I am delighted when the sun comes out.

    I was shopping for a new Gortex rain jacket with a built in hood yesterday and thinking this is standard issue.

    We love you and we love your blog for education about weather as well as so many other things.

    Please keep doing what you are doing and take your best guess. This is what has made you successful and I don't want you changing a thing.

    Eric

    ReplyDelete
  3. Wouldn't life be boring if there were no uncertainty? Of course, the engineer in me would be thrilled if the forecasts gave not only a % chance of rain but also a % chance of an accurate forecast. ie, 50% chance rain, 50% chance accurate forecast...

    ReplyDelete
  4. Can someone explain how an El Nino condition result in a split flow of the jet stream?
    Thomas

    ReplyDelete
  5. Any thoughts on forecasting snow to fill our aquifers?

    Cheers,
    Mike

    ReplyDelete
  6. Cliff,
    I was looking at some of the weather going on in Washington DC (very exciting, even from afar) and I noticed this map. What the heck is going on? Is that a tornado? A UFO? Something else that's easy to explain? :-)

    Screenshot of weirdness in NM/TX

    I wouldn't worry too much about last weekend. It seemed like a mixed-up kind of weekend and the weather's been so mild, it just feels like that lovely changeable spring time stuff, you know April showers that bring May flowers... it's only odd because it's February.

    ReplyDelete
  7. Amusingly I was using the WRF-GFS model in exactly this way on Friday to plan a birding trip with my wife to the Skagit flats. I ended picking Sunday not Saturday for the reason you describe (mostly to do with cloud cover and precip).

    I wanted it to be a bit convective (so sunny) to see more raptors in the air soaring.

    My wife wasn't too thrilled when we didn't get up early on Saturday and the day eventually turned out great. Then Sunday we got up early and ... nothing worth driving an hour for.

    We did get up there on Monday after a bit of weather radar nowcasting that convinced me the fog was going to break in the Skagit (convective showers are good for that).

    But point taken: look at the ensembles not a single model.

    ReplyDelete
  8. JewelyaZ said "I was looking at some of the weather going on in Washington DC (very exciting, even from afar) and I noticed this map. What the heck is going on? Is that a tornado? A UFO? Something else that's easy to explain? :-)"

    One erroneous data point (very much higher than the rest of the data ... 42.2 ... you can make it out compared to the 1 or 2 around the rest of the map) and a wrap around spectrum contour coloring scheme is just making it look weird (this is not a standard single spectrum coloring).

    It's just a spike (bad data in either this or the previous reading ... as these are pressure differences) at that location that's getting dressed up by the odd coloring scheme.

    ReplyDelete
  9. Daniel, as you may know, NOAA does give a forecast confidence number (1-5, poor to excellent, I think) for their 6-10 and 8-14 Day Outlooks. I WOULD like to see something similar along with each forecast period.

    ReplyDelete
  10. Found another indicator of just how warm it has been. Found a flea on my normally flea-free dog. Time to bust out the diatomaceous dust and penny royal!

    ReplyDelete
  11. I've never felt like local forecasters could beat the baseline: forecast clouds and showers every single day.

    ReplyDelete

Please make sure your comments are civil. Name calling and personal attacks are not appropriate.

Should I Let AI Write My Blogs?

I spend a lot of time thinking about and writing my blogs.   Could AI apps do the work for me?   Could any of you tell the difference? Well,...