Sunday, March 18, 2012

The U.S. Has Fallen Behind in Numerical Weather Prediction: Part I

 Part II found here.

It's a national embarrassment.  It has resulted in large unnecessary costs for the U.S. economy and needless endangerment of our citizens.  And it shouldn't be occurring.

What am I talking about?   The third rate status of numerical weather prediction in the U.S.  It is a huge story, an important story, but one the media has not touched, probably from lack of familiarity with a highly technical subject.   And the truth has been buried or unavailable to those not intimately involved in the U.S. weather prediction enterprise. This is an issue I have mentioned briefly in previous blogs, and one many of you have asked to learn more about.  It's time to discuss it.

Weather forecasting today is dependent on numerical weather prediction, the numerical solution of the equations that describe the atmosphere.  The technology of weather prediction has improved dramatically during the past decades as faster computers, better models, and much more data (mainly satellites) have become available.
Supercomputers are used for numerical weather prediction
U.S. numerical weather prediction has fallen to third or fourth place worldwide, with the clear leader in global numerical weather prediction (NWP) being the European Center for Medium Range Weather Forecasting (ECMWF).  And we have also fallen behind in ensembles (using many models to give probabilistic prediction) and high-resolution operational forecasting.  We used to be the world leader decades ago in numerical weather prediction:  NWP began and was perfected here in the U.S.  Ironically, we have the largest weather research community in the world and the largest collection of universities doing cutting-edge NWP research (like the University of Washington!).   Something is very, very wrong and I will talk about some of the issues here.  And our nation needs to fix it.

But to understand the problem, you have to understand the competition and the players.  And let me apologize upfront for the acronyms.

In the U.S., numerical weather prediction mainly takes place at the National Weather Service's Environmental Modeling Center (EMC), a part of NCEP (National Centers for Environmental Prediction).   They run a global model (GFS) and regional models (e.g., NAM).

The Europeans banded together decades ago to form the European Center for Medium-Range Forecasting (ECMWF), which runs a very good global model.  Several European countries run regional models as well.

The United Kingdom Met Office (UKMET) runs an excellent global model and regional models.   So does the Canadian Meteorological Center (CMC).

There are other major global NWP centers such as the Japanese Meteorological Agency (JMA), the U.S. Navy (FNMOC), the Australian center, one in Beijing, among others.   All of these centers collect worldwide data and do global NWP. 

The problem is that both objective and subjective comparisons indicate that the U.S. global model is number 3 or number 4 in quality, resulting in our forecasts being noticeably inferior to the competition.  Let me show you a rather technical graph (produced by the NWS) that illustrates this.  This figure shows the quality of the 500hPa forecast (about halfway up in the troposphere--approximately 18,000 ft) for the day 5 forecast.  The top graph is a measure of forecast skill (closer to 1 is better) from 1996 to 2012 for several models (U.S.--black, GFS; ECMWF-red, Canadian:  CMC-blue, UKMET: green, Navy: FNG, orange).  The bottom graph shows the difference between the U.S. and other nation's model skill.

 You first notice that forecasts are all getting better. That's good.  But you will notice that the most skillful forecast (closest to one)  is clearly the red one...the European Center.  The second best is the UKMET office.  The U.S. (GFS model) is third...roughly tied with the Canadians.
  
Here is a global model comparison done by the Canadian Meteorological Center, for various global models from 2009-2012 for the 120 h forecast.  This is a plot of error  (RMSE, root mean square error) again for 500 hPa, and only for North America.  Guess who is best again (lowest error)?--the European Center (green circle).  UKMET is next best, and the U.S. (NCEP, blue triangle) is back in the pack.


Lets looks at short-term errors.  Here is a plot from a paper by Garrett Wedam, Lynn McMurdie and myself comparing various models at 24, 48, and 72 hr for sea level pressure along the West Coast. Bigger bar means more error.  Guess who has the lowest errors by far?  You guessed it, ECMWF.


 I could show you a hundred of these plots, but the answers are very consistent.  ECMWF is the worldwide gold standard in global prediction, with the British (UKMET) second.   We are third or fourth (with the Canadians).   One way to describe this, is that the ECWMF model is not only better at the short range, but has about one day of additional predictability:  their 8 day forecast is about as skillful as our 7 day forecast.   Another way to look at it is that with the current upward trend in skill they are 5-7 years ahead of the U.S. 

Most forecasters understand the frequent superiority of the ECMWF model.  If you read the NWS forecast discussion, which is available online, you will frequently read how they often depend not on the U.S. model, but the ECMWF.  And during the January western WA snowstorm, it was the ECMWF model that first indicated the correct solution.  Recently, I talked to the CEO of a weather/climate related firm that was moving up to Seattle.  I asked them what model they were using:  the U.S. GFS?  He laughed, of course not...they were using the ECMWF.

A lot of U.S. firms are using the ECMWF and this is very costly, because the Europeans charge a lot to gain access to their gridded forecasts (hundreds of thousands of dollars per year).  Can you imagine how many millions of dollars are being spent by U.S. companies to secure ECMWF predictions?  But the cost of the inferior NWS forecasts are far greater than that, because many users cannot afford the ECMWF grids and the NWS uses their global predictions to drive the higher-resolution regional models--which are NOT duplicated by the Europeans.  All of U.S. NWP is dragged down by these second-rate forecasts and the costs for the nation has to be huge, since so much of our economy is weather sensitive.  Inferior NWP must be costing billions of dollars, perhaps many billions.

The question all of you must be wondering is why this bad situation exists.  How did the most technologically advanced country in the world, with the largest atmospheric sciences community, end up with third-rate global weather forecasts?   I believe I can tell you...in fact, I have been working on this issue for several decades (with little to show for it).  Some reasons:

1.   The U.S. has inadequate computer power available for numerical weather prediction.  The ECMWF is running models with substantially higher resolution than ours because they have more resources available for NWP.   This is simply ridiculous--the U.S. can afford the processors and disk space it would take.  We are talking about millions or tens of millions of dollars at most to have the hardware we need.   A part of the problem has been NWS procurement, that is not forward-leaning, using heavy metal IBM machines at very high costs.

2.  The U.S. has used inferior data assimilation.  A key aspect of NWP is to assimilate the observations to create a good description of the atmosphere.   The European Center, the UKMET Office, and the Canadians using 4DVAR, an advanced approach that requires lots of computer power.   We used an older, inferior approach (3DVAR).  The Europeans have been using 4DVAR for 20 years!   Right now, the U.S. is working on another advanced approach (ensemble-based data assimilation), but it is not operational yet.

3.  The NWS numerical weather prediction effort has been isolated and has not taken advantage of the research community.   NCEP's Environmental Modeling Center (EMC) is well known for its isolation and "not invented here" attitude.  While the European Center has lots of visitors and workshops, such things are a rarity at EMC.  Interactions with the university community have been limited and EMC has been reluctant to use the models and approaches developed by the U.S. research community.  (True story:  some of the advances in probabilistic weather prediction at the UW has been adopted by the Canadians, while the NWS had little interest).  The National Weather Service has invested very little in extramural research and when their budget is under pressure, university research is the first thing they reduce.  And the U.S. NWP center has been housed in a decaying building outside of D.C.,one  too small for their needs as well.  (Good news... a new building should be available soon).

4.  The NWS approach to weather related research has been ineffective and divided.  The governmnent weather research is NOT in the NWS, but rather in NOAA.  Thus, the head of the NWS and his leadership team do not have authority over folks doing research in support of his mission.  This has been an extraordinarily ineffective and wasteful system, with the NOAA research teams doing work that often has a marginal benefit for the NWS.

5.  Lack of leadership.   This is the key issue.  The folks in NCEP, NWS, and NOAA leadership have been willing to accept third-class status, providing lots of excuses, but not making the fundamental changes in organization and priority that could deal with the problem.  Lack of resources for NWP is another issue...but that is a decision made by NOAA/NWS/Dept of Commerce leadership.

This note is getting long, so I will wait to talk about the other problems in the NWS weather modeling efforts, such as our very poor ensemble (probabilistic) prediction systems.  One could write a paper on this...and I may.

I should stress that I am not alone in saying these things.  A blue-ribbon panel did a review of NCEP in 2009 and came to similar conclusions (found here).  And these issues are frequently noted at conferences, workshops, and meetings.

Let me note that the above is about the modeling aspects of the NWS, NOT the many people in the local forecast offices.  This part of the NWS is first-rate.  They suffer from inferior U.S. guidance and fortunately have access to the ECMWF global forecasts.  And there are some very good people at NCEP that have lacked the resources required and suitable organization necessary to push forward effectively.

This problem at the National Weather Service is not a weather prediction problem alone, but an example of a deeper national malaise.  It is related to other U.S. issues, like our inferior K-12 education system.  Our nation, gaining world leadership in almost all areas, became smug, self-satisfied, and a bit lazy.   We lost the impetus to be the best.   We were satisfied to coast.    And this attitude must end...in weather prediction, education, and everything else... or we will see our nation sink into mediocrity.

The U.S. can reclaim leadership in weather prediction, but I am not hopeful that things will change quickly without pressure from outside of the NWS.  The various weather user communities and our congressional representatives must deliver a strong message to the NWS that enough is enough, that the time for accepting mediocrity is over.  And the Weather Service requires the resources to be first rate, something it does not have at this point.

Part II will discuss the problems with ensemble and high-resolution numerical weather prediction in the U.S.

28 comments:

Kolya said...

Would it make more sense (from a financial perspective) overall to use a third party (specifically Amazon EC2) for the compute infrastructure?

mdeh said...

Cliff, Thanks for bringing this to our attention. Politics is funny...you never know when or how an article catches fire and starts the process of change. I have often likened our country to an elephant. It takes a lot to get an elephant to move...but once it starts, watch out. I hope your article will be the catalyst to get that elephant to move.

Justin Wilkerson said...

A very interesting article Cliff. Thanks for posting this.

I think it is good for people to understand why forecast can be so off nowadays.

You mentioned this, but didn't quite say it directly, but the problem is infrastructure. It is the problem our nation is facing in many areas. We are rather large nation (compared to places like European Countries, Japan, etc) with lots of outdated infrastructure and it is costing a lot of time and money to upgrade it all.

Of course as you mentioned for the improvement of NWP in our nation it really comes down to getting more advanced supercomputers with better data assimilation, while expensive I feel this would be far cheaper than many other infrastructure upgrades the U.S. is facing.


On a bit of a side note: I noticed that every single model you shown in the first graph (the 500hpa comparison) has what appears to be a seasonal variation. Oddly it looks like this variation occurs in the summer... do we actually find one portion of the year easier to forecast than another?

Glenn said...

It's pretty obvious, Professor Mass. The U.S. is an Empire in decline, and our failures in science and education are simply symptoms of that decline. The causes are another matter, not addressable here, and not reversible in any case.

Glenn

Jeffery said...

From the way the situation is described here, it sounds like it's just a matter of money. We could spend tens of millions of dollars to upgrade equipment for more advanced modeling, and have something about equal to Europe, or we could just keep renting from them. To the author it seems like a matter of national pride, but it might make more sense financially to let the Europeans do what they do best, like make clocks, look sexy, and forecast the weather.

Polistra said...

If NOAA spent less money on the Carbon Scam they'd have more money available to forecast REAL WEATHER.

I have no sympathy for them.

wanne1 said...

I'm not surprised since some members of Congress think they could eliminate the NWS as a cost cutting measure and rely on weather.com for forecasting. It's really a national security issue. How many functions can we outsource to ther countries before we fall irrevocably behind in the technology and we're getting our weather forecasts from China? Thanks for bringing up this issue.

richard583 said...

I'm not seeing the word "Congress" in this story, so far. (If mentioned here above, within one comment in response to it.) — This, whether with respect to its dragging its heals, or either otherwise, being lobbied, or better perhaps, pressed to "hear" arguments in favor of improving this situation. The whole idea, even prospect, would appear also, to be a good candidate for a well written, both concise and informative petition to sign onto. Good luck. ... Never check 'em myself. As a matter of general course.

Unknown said...

"We were satisfied to coast.    And this attitude must end...in weather prediction, education, and everything else... or we will see our nation sink into mediocrity."

I could not agree more. We have lived in a period of prosperity from the work of our parents and grandparents. There is someone somewhere working right now to do it (whatever it is) better, faster and for less. We can - we have to do better.

Thanks for the topic Dr. Mass.

GaryP. said...

So if computation power is an issue, why not use the same technology as the guys for SETI did and harness all those lost cycles via a screen saver? Or do the models only run on "big iron?"

Kate said...

Cliff, as an atmospheric modeler at EMC I first must thank you for bringing this issue to the forefront. I must point out, though, that our lagging behind is not from lack of want but directly from lack of funding, as you pointed out. We have had improvements prepared for years but lack the computing power (money) and man-power (money) to do so.

We also can't compete with the European center because they do not have the time constraints that we do...meaning our model output has to be out at a specific time without fail. Also, the ECMWF model only runs twice per day (GFS 4x) and is able to wait for more data to arrive.

I work with many talented, hardworking individuals at EMC who constantly have to deal with folks asking them why we aren't better when we know why...money. The government in this country yells at the NWS when forecasts aren't perfect but they do not understand that cutting our budgets is part of the problem, if not the main reason for the problem.

Lastly, we try to work with outside agencies but are stopped from doing so, again, because of budget constraints. I was recently barred from a project because those asking for the work didn't have money to fund it. If the US government put as much money into NOAA and the NWS as the European governments do we could compete as we used to.

FYI, the Hybrid EnKF data assimilation system will be operational next month and looks very promising! :-D

Again, thank you very much for bringing this unfortunate issue to light but please know that the problem is at the top, not at the bottom where all of the work is actually done.

P.S. we at NCEP are very much looking forward to our new building (coming this summer!), which is 10 years overdue because of government funding. There it is again...funding issues. :-(

Westside guy said...

This is a tangent, I suppose, but - what is the CDAS model that's so significantly worse than the others?

David Burch said...

Thanks very much. Very interesting. Will pass this link on to our marine weather students.

Scott said...

I did hear at the FOS meeting in New Orleans that the new Kalman Filter initialization scheme that NWS is implementing this year (already?) is supposed to boost them to a clear 3rd place, or maybe close to 2nd. Still behind ECMWF. Oh, I see Kate mentioned that upthread.

I think this blog post on GOES-15 WV issues says it all. Notice that the ECMWF is about to start using GOES Imager data in their global models. NCEP only uses Sounder data. And I get the feeling that they use it grudgingly.

Jesse said...

I believe that the CDAS was a "frozen" model used as a "constant" for the purposes of measuring the others. This is an excellent article and a good companion to one I wrote in 2009 entitled "Exposed: Secrets of Weather Forecasting Models" http://ow.ly/9LIfw

Tony said...

Cliff, you relate the problem to our K-12 Malaise. Yes, probably. In both cases, entrenched and often unionized interests prevent disruptive technologies and approaches from catalyzing rapid improvement.

My observation is you tend to lean pro-big-government, pro-union, and against the forces that tend to enable innovation.

Any chance you'll reconsider your political philosophies to bring the effects closer to your desires?

Unknown said...

Maybe we should just join the Europeans, and form a global modeling system...they're already ahead of us, it might be more economically efficient (no duplication), and realistically this is a global issue.

And, to the guy who said that "big government" and "entrenched unions" are the problem, it should be noted that we had more of that (lots more, in fact) during our golden era of innovation then we do now. Plus, the Europeans aren't exactly anti-union and anti-big government.

br549 said...

Great writeup Cliff .. look forward to part II .. domestic short-term NWP forecast skill is strategic for U.S. life and commerce. Therefore, it cannot be outsourced. We must regain leadership in this arena.

If the NWS and in this case NCEP had a Board of Directors: heads would have rolled a long time ago; if they had a stock: it would be $ 2.00 a share; if they had a balance sheet: they would be on the brink of extinction/bankruptcy.

This is a nearly complete failure against their most fundamental competitive mission. How can anyone explain that away?

Super said...

You see the problem captured by folks like Tony. Sorry Tony, but it's an example of the blame game that goes on and on with science funding tossed out. It's those evil unions and the liberal agenda. As another poster already pointed out, we were on top when those unions were more powerful. But history, and science for that matter, does not matter and will not be allowed to change some folks' reality. Science and funding it should not be partisan, but sad to say it has become so. Even for relatively small amounts of $$$, it's hard to keep investment up. See the recent cuts to tsunami warning and education (just a bit relevant to our coast line). Nothing will change until the political log jam of the past 10+ years breaks. Until then it will just be blame, blame, blame, with critical thinking checked at the door.

Unknown said...

I keep hearing that computer models already exist that can predict temperatures for the entire planet within a few degrees for decades into the future. Further, our most intelligent and best educated citizens are so confident in the accuracy of these models that they are diligently working to change our entire energy economy and tax structure to alleviate bad effects of the predicted temperatures. Perhaps one can be excused for thinking that we have plenty of computer forecasting technology already and don't need to spend any more money.

bubba0077 said...

@Unknown: you are conflating (whether intentionally or ignorantly) weather prediction with climate prediction. Climate is a description of the distribution of weather. If you think of it like dice, climate is the shape and values on the dice, whereas weather is the result of any particular dice roll. Both have substantial economic impact (billions of dollars per year).

Climate models have shown substantial skill in reproducing climates, but these have no more predictive power over knowing what weather will occur on a particular day in the future than knowing a die has six sides will help you predict which side will be face up after the next roll. Weather prediction, even with perfect models, is inherently limited to a few weeks.

As your comment demonstrates, ignorance of this difference has caused great harm to both areas. Climate research is dismissed because people remember that cold weather last week, or that weather forecast that was wrong. Weather forecasting gets cut because people fight against climate change research or think weather forecasting is solved because the climate models are considered good.

Anonymous said...

I'm surprised that no one has suggested the obvious -- why don't the USA and Canada pool their resources to develop a better model? If Europe can do it, then why not us? The way I see it, the days of America leading the world alone are petty much over, and not much chance of ever coming back. We can either accept that fact and make smart alliances or slowly descend into irrelevance, which is already occurring.

Mike said...

Great article. I did my MS at Colorado State and can attest to the great research that's occurring in the US. Sad to see barriers in bringing it to fruition in the operational world.

exoceano said...

Cliff - these are good points butt he systems you're comparing are not really on a level playing field. You need to include that ECMWF does not have the same level of service requirement as the US NCEP and FNMOC. ECMWF is medium range forecasting and our OPERATIONAL centers have a requirement to push forecasts out before the next Ob window. yes there are issues with not enough computational resources, etc. There are initiatives to leverage the National capability in a more comprehensive way and these efforts should continue to pursue more and new capability.

oklabob said...

Nice article. Note that although ECMWF doesn't have the same operational requirements as the other centres, Met Office, JMA and Meteo-France do have roughly the same time constraints as US operational centres, and are all doing very well in comparison. Of course the rankings change whenever a centre gets a new supercomputer or has a major change to its algorithms -- this was seen with the introduction of 4D-Var and then Hybrid-Var at various operational centres.

Bill said...

Very interesting article!

Darrin Harr said...

Excellent read!

anna said...

Cliff, this is a very interesting blog. Many European forecasters use NCEP model: it is good and easily available.

In one of the comments above, Kate mentioned that the EC model does not have time constraints. This is not entirely true: ECMWF data needs to be delivered on time every single day to the Meteorological Services of the countries that are contributing to its budget.

Looking forward to reading your part II
thanks again for the interesting post!