February 10, 2014

When Forecasts Go Wrong.

During the past week, forecast after forecast dealing with local snow has gone wrong.  On Thursday, Portland was hit hard even though the forecast a day before called for a 20% chance of snow showers.  On Saturday, the previous day's prediction  kept snow south of Seattle and even Saturday afternoon official forecasts (and this blogger) thought that the snow would not reach Seattle until late evening.   We were wrong.

Although no one likes to make a poor prediction, forecast busts like this week, when our most advanced tools fail, are what makes weather research fun.  This was not a human failure nor a mistake:  it was a demonstration that our science is still inadequate and our technology requires improvement.  It means future employment for folks like myself who spend much of our time researching weather systems and improving numerical models of the atmosphere.


It was thus with some amusement, that I and others in my fields have gotten some spicy comments and emails, some calling for apologies to those we deceived, others suggesting that these were easy forecasts if one only looked at the radar, or that we were somehow lazy and indifferent to the obvious.  Perhaps the most strident message I received was one demanding I stop teasing snow-hype-king Jim Forman of KING-5 TV.  Never!

Why do weather forecast models fail?

There are at least three reasons:

  1. The description of the atmosphere, the starting point of the simulation called the initialization, is flawed.
  2. The physics of the model, how basic processes like radiation, clouds and precipitation are described, are flawed.
  3. The forecasting problem is not possible considering the inherent uncertainties of atmospheric flows and the tendency for errors to grow in time.

Picture courtesy of KING TV

In my blog of last week, I noted that the nature of the snow forecasts this week were particularly hard.  A frontal zone, separating cold air over Washington and warm air over California, was stuck over Oregon.  A series of frontal disturbances or frontal lows forming over the ocean were moving eastward along the front.   These disturbances were small in scale, fast moving, and developing offshore where data was sparse. Thus, it was a very hard forecast, with the models unable to get a consistent solution.

And ALL the models were failing, including the state-of-the-art global European Center Model, the US. GFS, the US NAM, the UW WRF, the UW ensemble system, and the highly capable NOAA Rapid Refresh and HRRR models.    Extrapolating radar images might give you and hour or two lead time, but is useless for forecasts a day before.

For Saturday's snow that hit Seattle, model failures were profound.  The UW high-resolution WRF model's prediction of the snow, for the run starting at 4 AM Saturday, did not have snow over Seattle at dinner time (see graphic for 3-h snowfall ending 7 PM Saturday).  In fact, it kept the snow south of Olympia at that time.
In contrast, surface observations and NWS radar (at 7 PM, see below), showed that the snow band reached Seattle, about 100 miles north of the forecast position.  The European Center model did even worse ,and the National Weather Service short-range ensemble system (SREF)  did not suggest the northward movement (not shown).


Now one thing meteorologists do to determine whether to believe a model forecast is to check the consistency of the models...if they are all on the same page, one has more confidence in the forecast.  In this case, the models were pretty much uniformly holding the snow back from Seattle in the early evening.

Another test was to evaluate the initial conditions of the simulation or the fidelity of the early part of the forecast.  Here is the forecast for 10 AM on Saturday of sea level pressure, winds, and lower atmospheric temperature.  You see the low offshore of Oregon?

Here are observed low level winds at virtually the same time observed by a satellite sensor called a scatterometer (measures wind speed and direction by how microwave radiation is scattered off the waves). If you look carefully you will see the swirl of winds associated with the low...the model's position and shape for the low looks good.


And the track and amplitude of the simulated low continued to match observations later in the afternoon.

Perhaps most disappointing of all is the failure of the NOAA Rapid Refresh systems, designed to give good forecasts for the next 15-18 hours.  These system are run every hour, starting with the assimilation of a huge quantity of local data.  Here is the High Resolution Rapid Refresh (HRRR) forecast of 1-h snowfall valid at 6 PM for a simulation starting at 10 AM.  Way too far south.

And here is the total accumulation from HRRR through 1 AM Sunday....bad news--no snow over Seattle at all.  The HRRR system got better during the afternoon, but never got the timing right.  This is a real disappointment since HRRR often does very well in short term prediction.


So we have a mystery on our hands, at least until analyze the situation better.  Was there some subtle feature aloft that was not properly initialized in the simulations?  Was there an error in describing the moisture distribution offshore?  Was it an error in the model moist physics (cloud and precipitation description)?

Some detective work is needed.

As Sherlock Holmes said:  "The game is afoot"


16 comments:

  1. I'm still digging out from about six inches of wrong down here in Portland. :)

    ReplyDelete
  2. Do the models take into account the number of people wishing for snow and how hard they're wishing? ;)

    ReplyDelete
  3. In the Sumner area this morning, the roads were fine. However the snow had not completely melted - our original three or four inches of snow had melted down to about an inch remaining, sitting on top of a nice slushy layer. This made walking on grass and on sidewalks very treacherous! I ended up walking to the train station on the road rather than the sidewalk because the roads were in good shape (and the downtown roads aren't exactly full of cars in the early AM).

    This evening, though, the snow was completely gone. I'm glad for the warmup, but I do miss the snow.

    ReplyDelete
  4. This is the first time I've ever seen models from 7 days before were more accurate than models from the day before. haha

    ReplyDelete
  5. Chip DF said it--- and I've seen it happen a number of times--- we get a forecast 4-7 days out, a few days later it changes, but then we end up getting what was originally forecast. The early forecasts did say a possibility of significant snow Saturday, changing to rain with the approach of a warm front. That sort of happened, but the snow came separately, and the warm front was delayed. So that forecast wasn't wholly correct but did get the snow-on-Saturday part. '

    When was it apparent by radar that the "1/2" tops" forecast was wrong? 4 or 5PM? I wasn't watching radar, and it was only apparent to me when we had an inch already at 9PM and radar showed it would continue for 3 hours more.

    I also noticed the heavy snow earlier in the week in Portland also seemed to drift farther north than expected.

    ReplyDelete
  6. It would be great if forecasters could at least share their lack of confidence in these particular forcasting conditions. I think too many people trust short term forcasting to be very reliable and locally that does not prove very true. How about "there are a couple of possibilities this weekend...." and forecasters as least give the impression that its more of a guess at times. Greg in Gig Harbor

    ReplyDelete
  7. AS @schazjmd pointed out, never underestimate the power of 10,000 snow dances all occurring in the Seattle area. I was so happy to see it finally snow here. Would have preferred 6-8" and a cold front moving in after, but will take what we get.

    ReplyDelete
  8. I think the forecasts in such situations (which were known to be difficult, and for which the models had all been messing up for several days now) need to have some wording as to their uncertainty and low quality.

    Instead of confidently calling it a "minor snow situation", say something like "this is apparently a minor snow situation, but this sort of thing is generally hard to pin down well and has been particularly hard this week, so it is prudent to be on guard for significantly more snow".

    I spent an unexpected night in the city (didn't want to ride my bike back to the ferry terminal on slippery streets on the dark). By 7PM it was obvious that was going to happen: nearly an inch (i.e. more than the maximum forecast for the whole night) had already fallen where I was (north Beacon Hill), it was snowing continuously (no spotty showers this time), and the radar showed a nice big precipitation shield continuing to move in (at least several more hours of steady snow).

    ReplyDelete
  9. The ECMWF was the only model that nailed the initial snowstorm (last Thursday afternoon) in the Portland area. The 12z on the 5th had the low coming right into NW Oregon on the afternoon of the 6th. We just didn't necessarily believe it. Then that evening it was the only one to actually pound us with snow. It did the best down here. After that though timing was off on the following two systems.

    ReplyDelete
  10. To take what Chip said a bit further, I often see the 6-10 and 8-14 day long-range outlooks as being more reliable than the 12-36-hr forecasts. BTW, they call for cold and wet. :)

    ReplyDelete
  11. While I'm glad you were wrong ... I still respect your forecasts over anyone else here. Thank you Cliff!
    We are all imperfect beings :)

    ReplyDelete
  12. Of all places, Snoqualmie Ridge got nothing but flurries through the weekend. I suspect the persistent winds out of the east dried the air out too much. These downslope winds were unfortunately the correct part of the forecast from your prior blog post, known as the so-called "snoweaters".

    The last time this occurred in a major way was the big December snow storm of '08. Although then we still got 6", but Seattle easily had a foot.

    ReplyDelete
  13. In reference to forecasters sharing their uncertainty, I find reading the forecast discussion at the NOAA website useful in evaluating the certainty of the forecast. For example, for tonight I see the following in reference to winds in Whatcom County: ".WITH A LESSENING RISK AS YOU MOVE NORTH PAST THE SAN JUANS
    AND FIDALGO ISLAND. HAVE UPGRADED THE WATCH TO A WIND ADVISORY IN
    THESE LOCATIONS. WESTERN WHATCOM COUNTY SEEMS TOO CLOSE TO THE LOW
    TRACK TO MAKE STRONGER WIND LIKELY. WESTERN WHATCOM TENDS TO KEEP A
    LIGHT EAST WIND IN THE PRE-FRONTAL ENVIRONMENT...AND THE 12Z NAM MOS
    GUIDANCE NEVER PICK THE WIND AT KBLI ABOVE 10 KT TONIGHT. IT IS NOT
    TO SAY THAT STRONGER WIND IS IMPOSSIBLE TONIGHT...A MORE NORTHWARD
    TRACK OF THE LOW OR UNEXPECTEDLY GOOD MIXING OF WINDS ALOFT COULD
    YIELD BRIEFLY STRONGER WIND...BUT IT SEEMS UNLIKELY. THEREFORE...
    CANCELLED THE HIGH WIND WATCH FOR WESTERN WHATCOM.

    ReplyDelete
  14. There is a nice weather app found in the Apple Store. It is free. It goes by the name NOAA Radar Pro-weather alerts and forecasts. I used it this weekend to watch the snow make its way into the Puget Sound region and will be using it to watch Atlanta, Ga. tonight. Enjoy!

    ReplyDelete
  15. It is apparent confirmation bias is alive and well here in the comments.

    ReplyDelete
  16. It is always nice to get snow on a weekend, and one day for the kids to enjoy it. As happened this past weekend.

    It is easy to forget how much joy snow brought to us (so many years ago) when we were children in this neck of the woods. Forecast, or not forecast. I didn't give a darn back in the fifties...

    My grandkids were soooo excited Sunday morning.

    ReplyDelete

Please make sure your comments are civil. Name calling and personal attacks are not appropriate.