Who provides the most skillful forecast?
Online providers such as the WeatherChannel (weather.com) or the National Weather Service? How does this answer vary across the US? And how good a forecast is persistence--just predicting today what happened yesterday?
Such questions will be considered in the blog.
There is, in fact, an online website that provides answers to such questions: forecastadvisor.com (see sample below). They evaluate temperature and precipitation forecasts for hundreds of cities considering 1-3 day forecasts. They define temperature accuracy is the percent of forecasts within 3F of observed. Precipitation accuracy is the percentage of correct precipitation forecasts (precipitation versus no precipitation).
Examining the statistics for 2023 turns out the most skillful forecasts for each city was from the private sector (see table below), with weatherchannel/weather.com (WC) typically in first place, except for a few cities where Microsoft (MS) was the best. The National Weather Service is not the best anywhere.
I thought it might be interesting to plot on a map the difference in percent correct between the first place finisher and and the National Weather Service (shown below). The larger the number, the further behind is the National Weather Service, with numbers ranging from1.6 to 11.5 %.
The Chicago National Weather Service office was the best...only behind by 1.6%. In contrast, the Boston, Houston, Cleveland and San Francisco offices did not do very well. Not so good in Seattle (8.7).
Why is the Chicago NWS office so good? It is not because the weather is boring there! Clearly, excellent staff and local leadership.
Interesting but I see a couple of flaws in this analysis.
ReplyDeleteIt's not fair to compare NWS (or any other forecast) to the best of all the others in each location. Depending on how many others there are, some may be more accurate just by luck. (Similarly, if there are 1000 similar mutual funds, any one fund will compare poorly to the best of all the others.) It would be more meaningful to compare all the forecasts nationwide, by averaging their results, averaging their ranking, or some other method.
Also, saying a forecast is "right" or "wrong" about precipitation is simplistic. If a forecast says 40% chance of rain, it's not "wrong" if it rains. The forecast is a probabilistic statement. It would be more meaningful to look at the % of the time it precipitated when a given forecast predicted 0%, 10%, 20%, ... 100%.
Given the chart shown, it certainly seems that WC is better than NWS, and MS probably is too. So I'm not doubting your conclusion. But the data is not really organized to prove that.
So, why do you think NWS is less accurate?
Bruce.... I don't think you are correct. The statistics are based on an entire year of verification....and thus is highly statistically significant. Furthermore, previous years had similar statistics. The results ARE averaged for an entire year. ...cliff
DeleteNWS forecast is consistently about three degrees too warm during the summer in Marysville.
ReplyDeleteGrew up in Chicago; how much does their accuracy have to do with the fact that the weather coming across the plains is very obvious and easy to track?
ReplyDeleteRight, there are no obstructions to complicate things. Also, given the Prevailing Westerlies, there are many reporting stations west of there to get data from. Would think all that "heads-up" information would be very valuable.
DeleteYou've compared these before, and it's always interesting. I expect that the number of "unofficial" weather stations pouring data online into Weather Undergound is much bigger than the number of "official" stations that NWS uses. It would follow that the density of 'data points' would make a difference, even if many of the amateur stations aren't perfectly calibrated. I wonder, "What forecast model software" do the services use?" And so on. Bob Dylan's song "Subterranean Homesick Blues" included the line, "You don't need a weatherman to know which way the wind blows" ...but of course, everyone who follows this blog VERY MUCH DOES look-to the forecasts, and all of those are based on observation. You, sir, focus scientifically on the actual "drivers" - what makes the wind blow :)
ReplyDeleteWhat Weather Channel lacks is forecast discussions, 3 day history, etc. I'll check the Private forecasts but can't do without NWS.
ReplyDeleteOn top of that all these private forecasts look at the NWS discussions or have access to them, for free anytime they want.
Deletemichael....this is not correct. these services are automated and don't review nws discussions..cliff
DeleteI think the analysis from forecastadvisor.com would be more convincing if they publicly released more than summary statistics. Note, too, that some of the top rated private weather services - AccuWeather, The Weather Company, and Foreca - are clients of Intellovations, LLC, which owns forecastadviser.com.
ReplyDeleteFrom the Intellovations website (https://forecastwatch.com/about-us/eric-floehr/) - "Eric [the owner] used his experience in the complete software development process – from inception to funding to launch – to transform his love of big data analytics and web-based solutions into a thriving business with a range of clients that include AccuWeather, The Weather Company, Foreca and more."
I think my go-to weather app is Apple's native Weather App. It comes with mac OS X and mac iOS. Mostly, their data is sourced from services like NOAA-NWS and Weather.com, but others as well.
ReplyDeleteI'm not necessarily looking for absolute accuracy each time I want to know what the weather may be like. Mostly, I'm looking for a general idea of what lies ahead for the day and maybe the next few days.
I do like the fact that their screens are clear of all of the clutter found in the online commercial apps. Clarity and web (or app) design is important to me. There's already enough distraction in life without all that useless stuff to navigate.
Most important to me when I'm looking for accuracy is when I set out for a long 6-7 mile walk. I generally don't want it to be pouring when I reach my distal point in the walk. Here, I think the Apple app's precip screen (zoomed close into my walking range) works well. The other app I think excels for this need is the UW Department of Atmospheric Sciences' radar app. It seems more current than the GOES West screens though I'm guessing the data may be coming from the same sources.
Some screen shots and the data sources for the Apple Weather app:
https://www.litterrocks.com/weather-charts-and-graphics
Also, a link to the UW's radar site: https://atmos.uw.edu/current-weather/northwest-radar/
Cliff, I’m curious to learn how this analysis applies to UW Atmospheric Sciences Department insofar as I tend to refer to the site’s Weather Discussion page, day in and day out. My experience has been that, since I don’t need “accurate to the degree/pressure/humidity/wind speed” forecasts, the morning and afternoon UW weather discussions keep us relatively well advised of climatological change.
ReplyDeleteCliff, two thoughts:
ReplyDeleteFirst, it would be interesting to learn how accurate are private sector forecasts of days 11-15, 16-25, etc. AccuWeather and others try to tell the public it's gonna be x with a y% PoP and a high of z° as if they're to put as much stock into day 45 as day 3. I'd bet most readers here know better, but there are folks who take that guess as gospel.
Secondly, late advancements to AI seem poised to sharply increase accuracy, precision, and longevity of forecasts. What effect do you think that might have on both public and private sector weather forecasting?
Great blog, BTW.