Tuesday, February 25, 2014

Where is the National Weather Service's New Supercomputer?

It is nearly a year since the U.S. Congress supplied the money for a  new cutting-edge National Weather Service weather supercomputer, using Superstorm Sandy supplemental funds.

The computer promised to greatly improve weather prediction in the U.S. and was cited as a "game changer" by the head of the National Weather Service.

It offered the U.S. a chance to finally catch up with or exceed  the state-of-the-art predictions of the European Center, resulting in saved lives, improved warnings, and  large economic benefits for the United States.

Now a year later, the computer has not even been ordered, while the the European Center has just secured a brand-new American computer to push the envelope of weather prediction far beyond that practiced in the U.S.

This blog describes this situation and calls on Congress and others to demand immediate remedy of this sad state of affairs.

U.S Inferiority in Numerical Weather Prediction Continues

By objective verification statistics (see below for an example) and the admission of National Weather Service leadership, the U.S. trails behind many other nations in global weather prediction,with groups like the

A verification of the five days forecasts for sea level pressure over the northern hemisphere reveals that the National Weather Service global model (the GFS) is in fourth place, behind the European Center (ECM), the UK Met Office (UKM), and the Canadian Met. Center (CMC).  The higher the numbers the better (1 is a perfect forecast).  Note the consistent high numbers for the ECM, (red dashed) with some major dropouts (e.g., Feb 22) for the US GFS model (black line).

European Center for Medium Range Weather Forecasting and the UKMET office consistently outperforming the U.S. operational numerical weather prediction center (the National Weather Service's Environmental Modeling Center, EMC).  Lately, the Canadian Meteorological Center has been superior as well.  As shown inthe above figure, the U.S. would not even get the bronze medal in weather prediction.

This inferiority was made clear to the general population during the period before Superstorm Sandy's landfall, when the U.S. models took several days to come into line with the highly accurate 6-8 day forecasts of the European Center model (discussed in my previous blogs here and here). More seriously, U.S. weakness in numerical prediction is not limited to global modeling:  for example, we are failing to run high-resolution ensemble forecasts that would allow far better prediction of severe thunderstorms and other local weather features.

Lack of Computer Power is Crippling U.S. Weather Prediction

As noted by many, including NWS Environmental Modeling Center staff, a major reason why the U.S. lags in numerical weather prediction is the lack of computer resources available to the National Weather Service. Today, the National Weather Service has two computers with peak performance of approximately  210 teraflops and 10,240 processors (Note:  1 petaflop equals 1000 teraflop, a teraflop is a trillion operations per second).   This US computer recently was upgraded from 80 to 210 teraflops, with a change in computer architecture from IBM proprietary to commodity chips.   In contrast, the European Center has just acquired TWO U.S.-made CRAY XC-30 computers, each with 80,000 processors.  My sources indicate that each EC computer has a peak throughput of 2-5 petaflops and a sustained 200 teraflop capability, dwarfing the computers the NWS has today. Translation:  the new European Center computer is approximately 15 times faster than the current National Weather Service computer.  And our computer has to do FAR more tasks.

Current Computer Resources (peak computational capacity in petaflops) of the European Center Versus the National Weather Service Environmental Modeling Center.  The NWS has far more modeling responsibilities than the EC

The funds from the Sandy supplemental funding would be enough to secure a computer from IBM with a peak performance of 2 petaflops, substantially less than the European Center computer.  And keep in mind that the European Center only has to do global weather prediction while the National Weather Service has vastly more modeling responsibilities, including high-resolution forecasting over a vast nation.   The U.S. really needs a 20-30 petaflop computer resource to provide U.S. citizens with state-of-the-art weather forecasting.  Such machines exist and are being used for climate prediction and simulations of nuclear explosions....do we have our priorities wrong?

This lack of computer power guarantees U.S. inferiority in weather prediction.  Some examples:

1.  The U.S. global model, the GFS runs with 26 km grid spacing, while the EC is at 16 km, and they plan to go to 10 km very soon.
2.  The U.S. does not have the computer power to run high-resolution ensembles (many forecasts) over the U.S., a necessity for prediction of thunderstorms and other local weather features.
3.  The U.S. has lagged substantially in the area of data assimilation, the technology of effectively using observations to describe the structure of the atmosphere...the starting point of all numerical weather predictions.
4.  The U.S.lacks the computer power for high-resolution global ensembles, which are critical for prediction of uncertainty and probabilities.  Such ensembles are also critically needed for weather data assimilation.

The effective use of a large computer system, like the European Center has, would greatly improve U.S. weather prediction, clearly saving lives and promoting the economic needs of the U.S. (e.g., agriculture, water resources, renewable energy).    U.S. companies now spend many millions of dollars to get the European Center model forecasts; they would not need to do so if the U.S. predictions were the best.

Why the Delay?

With such an acute need for enhanced computer power, you would think the U.S. government would have initiated the acquisition of new weather supercomputers immediately and just as quickly begin testing the improved weather forecasting models that would move to the new system.

If you thought that, you would be wrong.  Today, a year after the funds became available a new NWS supercomputer has not been ordered.  The reason given?  IBM's sale of its server division to Lenovo.

It turns out that NOAA made a very bad decision in their computer acquisitions.  They committed themselves for an extended period to buy only from IBM, thus losing the ability to shop around for the best deal during each replacement cycle.  And now IBM is planning to sell its server business to Lenovo, a Chinese company, and this is setting off all sorts of warning bells in the U.S. government, and specifically for the new weather computer acquisition.   If NOAA was not stuck with IBM they could go with other vendors, like CRAY, which sells U.S. machines that have no security issues (in fact, the European Center liked them so much they bought one!).

The worry about the Chinese server business is particularly ridiculous for a weather forecasting computer.  Does the U.S. government think the Chinese will turn off U.S. weather prediction in case of a conflict? Even if they could do that, it would make little sense.  Many Chinese groups download and use the use U.S. GFS weather model output.  If they somehow turned off our weather forecasts, we would always have the European Center, the Canadian model , the UKMET office model , and the forecasts from the U.S. Navy!  So it would not get them anything.
And there is something else you might not know of.  A huge proportion of the U.S. modeling community hails from China.  I do a lot of numerical modeling, and when I go to conferences and attend sessions on topics like weather data assimilation, it is evident that if the presenters changed language to Mandarin, the majority of the audience would be quite content.  China and the U.S. have a close partnership in weather prediction with profound interdependence.  They are not going to destroy our weather modeling capability.  And this far fetched threat should be balanced against a real threat.  In fact, not a threat, but a reality:  the inferiority of U.S. weather prediction, the cost of which is very real to each U.S. citizen.   There is all kinds of talk about extreme weather and climate change, yet we don't do the one thing that is guaranteed to afford substantial protection for our population from extreme weather:  better weather prediction.

A year ago, high NOAA and NWS administrators were flashing powerpoints with time lines showing that the new NWS supercomputer coming online during the second quarter of 2014.   Their latest presentations say nothing about upgrades during the next few years and talk about what they will be doing in 2018!  And there is NO sign of an active process of designing and testing the next generation models that could immediately be used on a far more capable supercomputer.    Just a continuation of the go slow approach that has kept U.S. weather prediction mired in the mud for so many years.

Recently, there have some changes in NWS leadership, with a particularly capable individual becoming head of NCEP (National Centers for Environmental Prediction).  There are some very capable modelers/scientists in NOAA/NWS.   But they need sufficient computer resources and they don't have them now.  And there is an acute need for a detailed strategic plan on how the U.S. will regain leadership in weather prediction...which they don't have.  It is time for Congress or the administration to end the foolishness with the computer contract, or to void the IBM contract and find another vendor.

An interesting possibility is for IBM to purchase a CRAY XC-30 and use that for the NOAA/NWS contract.

The European Center folks are just loving the stalled U.S. weather prediction effort.   As noted above, they are making out like bandits selling their model forecasts to U.S. and other countries...we are talking about many millions of dollars.   And I understand that European Center staff bonuses are partly based on how much they are ahead of the U.S.  So the current situation is like money in their pockets.  A year ago, I was at a meeting where I had a chat with the head of the European Center.   He had seen my blogs and told me that the U.S. should not worry about leading weather prediction since we were all in this together...or something like that.  Right.  Keep in mind that the European Center is never going to take care of most of the U.S. numerical weather prediction needs, like high-resolution national ensembles for thunderstorm prediction. And the money is rolling in for them as long as the U.S. is behind.

Why Not Strive for the Gold?

     In Olympic sports, the U.S. strives for and often achieves gold.  We do the same in so many areas of technology and science.  So why is the U.S., and particularly NOAA/Dept of Commerce management,

Why not the U.S.?
satisfied with bronze status, particularly when it costs the American people dearly in lost lives, injuries, and a multitude of economic costs?   How long will the political leadership of our country accept this situation?

Perhaps we need a bit of General Patton's attitude towards winning and leadership (click on picture)


snapdragon said...

Don't be too hard on the gov't, Cliff... they've been kind of busy lately. But I am sure that as soon as they figure out the health care system they will hurry up and get that new computer. Yup. It's next on their agenda. Any day now...

Aaron Bockelie said...

I hear there is a brand new government affiliated data center in Utah. They've got a great team, plentiful compute and storage available to collect manifold data points and assemble results in creative ways. There may be an issue sharing the reports with the general public, though.

Harrison said...

Cliff - What's this talk about a big snow storm in Seattle this weekend? 1 ft of snow showing as the NWS says. Is this a future blog post? Probably too far out to say I assume.

Cliff Mass said...

Too much uncertainty at this point...but yes, we will have cold air in place by this weekend...the question is precipitation...cliff

Unknown said...

Once again, my hopes are high for a snow event The GFS and Canadian runs seem to really want to give us a snow day on Monday. Let's hope they stay the course this time!

Anthony Soprano said...

The IBM server division sold to Lenovo doesn't make super computers. Servers and Super Computers are two different animals. Is the NWS planning on purchasing servers and using them to run new modeling software? Foolishness.

GaryP. said...

"The U.S. really needs a 20-30 petaflop computer resource to provide U.S. citizens with state-of-the-art weather forecasting. Such machines exist and are being used for climate prediction and simulations of nuclear explosions....do we have our priorities wrong?"

Well if we all die from a nuclear explosion or climate change who will care what the weather is?

Seriously CRAY is a Washington company, they have offices down in Pioneer Sq. Surely our Senators would like to see more business come to our state?

Time to make some phone calls.

Dan McGuan said...

System X is indeed used in supercomputers Anthony... and used by NOAA. IBM calls them "Intelligent Clusters". NCAR's Yellowstone Supercomputing Center took delivery of a System X based supercomputer last year called Yellowstone - a 1.5 PFLOPs system comprised of over 75,000 cores of a System X architecture called iDataPleX. Its the largest machine ever built specifically for "earth science", however, I'm not sure how frequently it will be used for weather modeling.

James Westbury said...

Cliff, is there any possibility of doing distributed computing a la SETI, cancer research, etc. for forecasting, or do the particulars of weather forecasting somehow preclude this?

Chris Upham said...

Cliff - Can you share the URL for the euro model? I enjoy reviewing the NAM and the GFS but have yet to find one for the euro. Thanks

Johannes Rexx said...

So perhaps it is time for the weather forecasting software to encompass more efficient algorithms that can produce better results on less compute power? Alternatively, perhaps it's time to take advantage of distributed computing such as Folding@Home and SETI?

Unknown said...

Welcome to corporate America!

Michael said...

Cliff- you should reach out to someone at Amazon Web Services. Perhaps the supercomputer the NWS needs is already built and operating, just waiting for them to use. In November Cycle Computing spun up a 156,314 core HPC on AWS that hit 1.21 petaflops.

Unknown said...

Michael hit the nail in the head. AWS offers computing power for cheap. I use them, many others do; heck, the CIA uses them. Why not NWS too? It's highly scalable, with little in the way of sunk costs or antiquation.

Cliff Mass said...

Let me answer some of the questions about alternate computer resources.

A distributed SETI-like system can't work because the processors need to talk to each other rapidly and that is not possible with a SETI architecture. And most cloud computing resources won't work because the nodes need to talk to each with very high bandwidth--like 40gbitper second or more. Most distributed nodes do not have such connectivity. Perhaps that will change...cliff

CNY Roger said...

Thanks for bringing this to my attention. I sent this to my senators and congressman this morning. I have been a meteorologist for 40 years and completely endorse the sentiments in this blog posting (http://cliffmass.blogspot.com/2014/02/where-is-national-weather-services-new.html) on the state of US weather forecasting computers. In brief, the National Weather Service forecasting capability is severely hampered by a lack of the latest and greatest forecast computing capability. As a result, US forecasters are relying on forecast products from Europe, the United Kingdom and Canada which are performing better. However, those forecast products do not provide information necessary to improve local emergency forecasts which still rely on the inferior US products. It is time for Congress to fix this problem as soon as possible. A good start would be getting the roadblocks out of the way that have prevented the purchase of advanced computers with Sandy supplemental funding money already approved.

Kevin said...

FOA of Chris -

My understanding is that the only place you can find the euro model is here:

Based on this, I don't see the warm up in Whatcom county that the forecast discussion mentions for Sunday....

Hindu said...

When and if we make the needed hardware upgrades, how does our software stack up vs the Euro?

Aaron M. said...

The big issue with foreign hardware is the risk of hardware backdoors being installed which could compromise the security of the devices. There's a lot of work being done in both the computer architecture and computer security fields to address this. A popular option is to use a Trusted Platform Module (TPM), a chip which secures the machine from the hardware up. If the US government would simply seal a prototype known to be acceptable with a TPM, they could farm the design off to SE Asia (where just about all hardware fabrication is done) and then use the US-built TPMs to verify the integrity of the returned product via remote attestation.

Your suggestion of purchasing a Cray machine is flawed. Cray uses commodity off-the-shelf (COTS) hardware for most of their components. These are primarily fabricated in SE Asia (usually Taiwan). At least when I was an intern there the only hardware they designed themselves was the networking hardware. And even that was fabricated outside of the country. The processors, memory sticks, microcontrollers, etc. were all commodity hardware. A few years back, Intel acquired their hardware division and while they're still based in Seattle, Cray machines now use Intel components and those from Intel's vendors, which are also fabricated abroad. So the rabbit hole goes a lot deeper than you think...

Michael Snyder said...

If you want the EURO and all its glory you need to go to ACCUWEATHER.com and get the professional account then plunk down 25$ a month to get all the high resolution runs over much of the planet.

Tons of good stuff, and the forecast model animator is a really nice tool. It allows you to compare and contrast all the models, with each other or against previous model runs.
There are more options I could ever list here.

StormCchaser said...

One annoying thing is that most of the European model output is not openly available - you have to pay for it. US models are freely available.

Michael Dross said...


What exactly hasn't been ordered? There is some confusion. The NWS did upgrade from AIX to x86/Red Hat based systems last year as you noted.


Is it the next upgrade slated for 2015 that is going to be delayed?

Michael Dross said...

Also, according to this article Australia's Bureau of Meteorology is looking to replacing its current supercomputer with a new one this year. It may be as large as 86,000 x86 cores each, if it's indeed 50% bigger than "Raijin" which would put it up there with the ECMWF in terms of computational horsepower.