January 10, 2017

Make U.S. Numerical Weather Prediction Great (Again)

There are many folks in my community (climate and weather scientists) that are apprehensive about the upcoming Trump presidency.  They worry about Trump's unfounded suggestion that climate change is a "Chinese hoax" and his appointments to major office of folks that question the threat of increasing greenhouse gas.  They are concerned, like I am, that Trump might slash U.S. government support of climate change research, the importance of which I described in a previous blog.

Could a bull in a china shop be helpful?

But lets consider an intriguing possibility.   An unconventional Trump administration could choose to do something important and helpful:  restore U.S numerical weather prediction (NWP)  to its rightful place as the best in the world.   And in doing so, immeasurably foster the public safety and economic vitality of the U.S.   And don't forget:  better weather prediction science and technology leads to better climate forecasts.

Trump wants to make American great again.  Fine.  Let him start with a key technology in which the U.S. has unnecessarily fallen behind:  U.S. numerical weather prediction, the central tool for all weather forecasting.   As I will describe below, U.S. NWP development and operations needs a "disruptive" influence that might reorganize an ineffective bureaucratic structure and a bureaucracy that has been content to be second or third best.

Behind and not catching up

In a number of blogs I have documented the inadequacies of U.S. operational numerical weather prediction, with the NOAA/National Weather Service (NWS) having primary responsibility.  First, the U.S. is behind in global weather prediction--lagging both other nations (e.g., the European Center, the British UKMET office) and far behind the inherent capabilities of the U.S.  And with all the talk and investments of the last few years, the U.S. is not catching up (as documented below).

Let me demonstrate this to you with two graphics.  The first shows the skill over the past month of the global forecasts at 500 hPa (about 18,000 ft) for the 6-day forecast for several global modeling systems.  The quantity shown is the anomaly correlation, with 1 being perfect.  The European Center (ECM) is best (.857), followed by the British (UKMET) with .840, the U.S. NWS (.814), and trailed by the Canadians (CMC) and the US Navy (.756).

But here is another way the U.S. (NWS) global prediction is behind--the number of drop-outs, periods of much lower prediction skill.  This plot shows anomaly correlation for 500 hPa for the Northern Hemisphere (20-80N)  five-day forecast during the past year for  the U.S. (GFS model, red) and the European Center model (ECMWF, blue).  ECMWF is nearly always better and you see frequent downward excursions of the U.S. GFS model, sometimes down to very low skill levels.  These are the drop-outs and indicate major loss of skill.  Very bad.


I could have shown you a similar plot two years ago and the story would have been the same.  We are simply not catching up.

For seasonal forecasts (out to 9 months), the NWS is running a seasonal climate model (the CFS) , even though it has no demonstrable skill over statistical methods (I am doing research on this right now).  But there seems little impetus to pull the plug on this non-productive effort or make the radical changes necessary to make seasonal or subseasonal (weeks to a few months) better.

The U.S. is profoundly lagging in other areas of numerical weather prediction as well.  Numerous workshops, reports, and meetings have called for the National Weather Service to run a high-resolution ensemble (many forecasts) over the U.S. to aid forecasting of thunderstorms and smaller scale features. They have not done so, while the research community (NCAR) has an experimental operational system in place.  Nor has the NWS been effective in developing and applying improved model physics (such as the microphysics of clouds) and its statistical post-processing (improving the forecasts with statistics) is based on simple 1960s approaches.
The unphysical mottled appearance of NWS GFS model precipitation forecast over terrain is a sign of a serious error in model physics.  The European Center forecasts do not have this flaw.

There are many other examples of U.S. numerical weather prediction falling behind the inherent capabilities and needs of the nation, but the above examples suffice.   And keep in mind that the U.S. spends more on numerical prediction than anyone else and has the biggest weather research community in the world.

The questions you must be thinking is:  why is the U.S., the past leaders in numerical weather prediction, fallen behind, and how can this be fixed?   

And I will ask another question:

Could folks with a very different perspective and unafraid to break a few eggs (the Trump administration) make some acutely needed changes in U.S. NWP that could set it on a much better course?

The Real Reasons Why U.S. Numerical Weather Prediction (NWP) is Behind

I have studied this question for years, published peer-reviewed articles on the subject, and have served (and am now serving on) advisory committees to the U.S. government and other entities.  I believe I know where the skeletons in the closets are found.  And I am at the point in my career that I can afford to take some risks in the hope of promoting critically needed change.   So here it goes.


Reason Number 1:   Awkward and Ineffective Organization and Bureaucratic Structures

It would be hard to think of an organizational structure that is more problematic than the one used by NOAA/NWS/US government NWP development and research.  The byzantine organizational structure of NOAA and the NWS are found below for reference.


But first a lot of acronyms.  Operational numerical weather prediction (NWP) is found in the Environmental Modeling Center (EMC) of  the NCEP (National Centers for Environmental Prediction), which is part of the National Weather Service (NWS), which is part of the National Oceanographic and Atmospheric Administration (NOAA).

Amazingly, folks with the main responsibility for developing the models are NOT in the NWS, which is responsible for operational NWP, but in NOAA (at the ESRL and GFDL labs).  So the folks who run the weather forecasting models (NCEP EMC) don't have control of the scientists developing the models.

And this has led to a LOT of problems.  For example, the NOAA ESRL folks spent a lot of resources on models that never went operational, antagonism between the model developers and the operational people has been rampant (although better recently), and other issues.  

But it is worse than that.  The decisions on future model use is decided upon by ANOTHER group in the NWS outside of NCEP, called OSTP (Office of Science and Technology Policy).     And the development of the statistical post-processing of model is not done by the modelers or in the same organization doing operational NWP (NCEP's EMS), but another group in OSTP (called MDL).    It is surprising ANYTHING is accomplished in this crazy quilt bureaucracy.

But it is worse than that.  The U.S. Navy runs its own independent global and regional weather prediction effort at FNMOC (Fleet Numerical Meteorology and Oceanographic Center), based on science/technology developed by the Navy's own research arm (MRL Monterey).  A mainly duplicative effort, except for a greater emphasis on the oceans.  And all this is done, even though the Navy's global model's verification scores are consistently inferior to the National Weather Services (see above for proof, FNO is the Navy).   I am amazed Congress hasn't noticed this.

Can it get even worse and wasteful?  You bet.  The US Air Force runs ANOTHER NWP center and they decided to use a foreign model (the UKMET office unified model), rejecting the previous use of the NWS model (GFS) and aregional scale WRF model developed by the academic community (NCAR, the National Center for Atmospheric Research).   This decision was so silly, that anonymous researchers in the US Air Force (AFWA, Air Force Weather Agency, now Detachment 7), made a very funny video about it, suggesting that even Hitler would have been upset about the wacky decision.  So the US Air Force thought the NWS global model was so poor, they had to run a foreign model.  Just stunning.

http://www.captiongenerator.com/37268/Hitler-learns-the-USAF-is-changing-weather-models

In short, huge amounts of U.S. resources have been wasted in funding duplicate national weather prediction entities, with all being suboptimal for state-of-the-science weather prediction.  Combined and organized in a rational way, they could be the best in the world. Separated in this separate efforts, they are seriously lagging behind.

Reason Number 2:  Isolation of Governmental NWP Development from the U.S. Research Community

Numerical weather prediction is perhaps the most complicated technology of our species, with billion dollar satellites, international observing systems, the biggest supercomputers, and complex data assimilation and modeling software encompassing millions of lines of code.  It considers and models physical phenomena from the molecular to the planetary scales.  Only the organized and coherent efforts of the large U.S. research community can push the envelope to limits of our knowledge and technology.

As noted above, U.S. efforts have been divided among many efforts in the Federal government.  But the problem is far more serious than that.  The National Weather Service model development has been isolated from the vast U.S. academic community, and thus U.S. operational prediction has not fully benefited from the research sponsored by Federal Agencies and others. The inability to take advantage of the work of the U.S. research community is sometimes called the "Valley of Death."  A not invented here syndrome has been a major characteristic of NCEP's Environmental Modeling Center.


There are two good examples of NOAA/NWS isolation from the academic community. In the late 1990s, NCEP EMC realized it needed a new regional model to replace the deficient Eta model.  Instead of using the model developed at the National Center for Atmospheric Research (NCAR, an organization of US academic departments in atmospheric sciences) and used by thousands of researchers (WRF, Weather Research and Forecasting Model), they developed their own (NMM), a model that was not only inferior but never used by the academic community.

More recently, the NWS realized it needed to replace its aging GFS global model.  Instead of joining the academic community in using the new, but proven MPAS model developed by NCAR, they decided to acquire a model from a NOAA group (GFDL).  Thus, the isolation continues, with stark implications for U.S. operational prediction.  Only by joining with the academic community can the NWS hope to develop a state-of-science modeling system.

Reason Number 3:  Lack of Strategic Planning

NOAA/NWS numerical weather prediction efforts not only suffer from an inherently poor organizational structure for operations and research, but they have functioned without an actionable strategic plan and specifically one with clear/concrete goals and time tables.  They are working on one now, but I have been worried about its lack of detail.  The NWS has begun to fund extramural research on NWP to a greater degree, a very good thing for which NWS Director Louis Uccellini deserves a lot of credit.  But without strategic and implementation plans and a clear sense of priority, much of this funding is not being used wisely.

Reason Number 4:  Inadequate Computer Resources

Two years ago, the National Weather Service had 1/10th the computer power of the European Center, yet the NWS had far more responsibilities (both global and regional/local modeling) that the European Center.  After outside complaints, Congress and the administrator improved the situation, increasing computer resources to equity with the European Center.  But such computer resources are still woefully inadequate.  Based on my own analysis and that of several committees I have served on, the NWS need 30-50 times more computer power that it has today (total of 100-150 petaflops capacity) to forecast in a state-of-the-art manner, with a huge benefit for the nation.   The cost?  Roughly 100 million dollars, the price of a few fighter jets.  What do you think would benefit U.S. citizens more?  Dropping redundant or ineffective models, would also release computer power (but that is a drop in the bucket to what is needed).  And our local friends are CRAY computer are ready to provide the hardware.


Good People, Mediocre Results

Let me make something clear.  Many of the folks in NOAA/NWS are competent and interested in providing the U.S. with cutting edge numerical weather prediction capabilities.   I know personally many of those in leadership/management positions and most are highly motivated and have the required technical knowledge.  They want to make things better and are taking some steps.  For example, the NOAA/NWS have sponsored a number of reports (by the National Academy of Sciences and McKinsey consultants) and several advisory groups.  Things may get a little better, but the U.S. will still fall further behind because the fundamental issues, many noted above, that are not getting addressed.   Some NOAA management types complain that the real problem are NOAA/NWS unions, who they claim are dragging their feet on change.  As noted above, the real problems are elsewhere.

The U.S. Congress knows that U.S. weather forecasting is lagging, with several hearings on the topic.  In December, Congress came close to passing the Weather Forecasting Improvement and Innovations Act that would have invested hundreds of millions of dollars to fix the problems.   Whoever wrote it meant well, and addressed a few of the issues, but they didn't appreciate the fundamental organizational issues note above.

So What is Needed?

The only way U.S. NWP development and operations will be improved permanently is by a radical restructuring of the governmental forecast enterprise. No single agency or department head has the power to correct the structural and resource deficiencies.   Only the Congress and the Administration can fix it, and they need to be willing to make some tough choices.

Restructuring US NWP will require folks with an intimate knowledge of the relevant technologies, as well as representation of the entire U.S. weather prediction enterprise (government, academic, private sector, users).   If I was the President, I would establish a blue-ribbon committee with such representation to come up with an analysis and concrete plan.  It requires a willingness to prune an overgrown tree, so that the remaining branches can be flourish.  It will require the development of concrete strategic and implementation plans and bringing together of the vast U.S. weather research and development community.

Trump wants to make the U.S. great again....perhaps, such perhaps, his administration might take advantage of their outsider status to make the changes that are so acutely needed to make U.S. weather prediction great again.



_________________________________
Northwest Weather Workshop

The Northwest Weather Workshop, the region's main gathering to discuss all aspects of Northwest weather, will take place on March 3-4, 2017 in Seattle at NOAA Sand Point.  There will be a special session of communicating forecast uncertainty during the first day.  More information on the meeting, as well as registration details, are  found at: https://www.atmos.washington.edu/pnww/

If you are interested in giving a talk at the meeting, please send me a title and short abstract by February 1.


23 comments:

  1. I like the scrambling eggs analogy. The duplication of effort at all levels of government is astounding and it sounds like weather forecasting is not immune to this affliction. Thank you for having the intestinal fortitude to share your thoughts and opinions on it.
    Les Williams

    ReplyDelete
  2. One thing that would help the entire climatology field is to allow prominent researchers such as Richard Lindzen and other voices (such as Judith Curry) a role in these discussions. The squelching of any discordant voices in the current mantra is not a signal of confidence of their assertions. Additionally, getting the government out of crony capitalistic schemes such as the carbon trading fraud and the Solyndra fiasco will only help the public perception of climatology in the long run.

    ReplyDelete
  3. One small point: How much significance is there between the 0.857 ECM skill and the 0.814 US NWS? If you do an anomaly correlation for each month for the past 36 months and do this for both the EMC and the US NWS, and then calculate the average skill level and 3-sigma errors from those individual skill determinations -- how much is the difference between 0.86 and 0.81 outside those 3-sigma error bars? This will give you a measure of how far you're modeling is really behind... Cheers!

    ReplyDelete
  4. Just out of curiosity, is the modeling still such that each cell only needs data from its adjacent cells? When I studied parallel processing back in the 70s, weather modeling was given as an example of SIMD architecture, ie single instruction, multiple data points, in that every cell could be doing the exactly same calculation at once. If so, I would suspect the current super computers are rather ill suited to the problem--they're running 10k+ Xeon multicore processors, which are spending most of their energy decoding instructions, managing pipelines, cache memory etc. I haven't looked at chip layout in quite a while, but I can't imagine the control logic got any less complicated. If the basics of modeling hasn't changed,you need most of your chip dedicated to computation, and very little in control...of course that would mean making your own chips for probably a very limited market.

    ReplyDelete
  5. Blah Blah Blah......you and the NWS have have so off the last week it makes your comments and especially your forecasts laughable. Right now in West Seattle we are getting high winds, with higher gusts,.......yet no one said anything about that. Snow is just outside of Olympia heading north.

    All I need at this point is my outside temperature gauge, a look at the radar, look out the window and I'm golden. Or maybe I should ask my dog???

    ReplyDelete
  6. According to Trump, models don't work. You need to go back to using couriers!

    ReplyDelete
  7. Correction! " Roughly 100 million dollars, the price of a few fighter jets" should probably be "Roughly 100 million dollars, the price of an F-35 ... without an engine."

    Citation: www.f35.com/about/fast-facts/cost (a Lockheed martin website)

    ReplyDelete
  8. I read this blog often, and I have learned a lot here. However, I am disappointed that you choose to discuss politics rather than science. I hoped you were above that.

    ReplyDelete
  9. Can you do a write up about the snow event in Portland. What was to be a dusting to 4 inch event turned out to be one of the biggest snowfalls in some time. I have 11-12 inches at my place in the west side of Portland.

    While playing with the dog in the middle of the park I swear I saw lightening. How did the models miss this?

    ReplyDelete
    Replies
    1. We got your 4 inches here in Eatonville, Washington and the dusting went as far north as Graham. We were supposed to get 1-3 inches Monday night but got not so much as a single flake. No mention of snow at all for last night and we end up with 4+ inches....go figure.

      Delete
  10. Cliff, absolutely love your detailed analysis on this! You have been hitting this for as long as I have read your blog. There is NO reason the US cannot and should not be the best in weather modeling and forecasting. This is why governmental bureaucracy drives me crazy! We have the money, we have the talent, we should be the best. I think you should forward this well written analysis to the powers at be until you get some results. I for one, am going to write my senator and congressman in hopes someone will listen! Btw, we have had flurries all night in SE Auburn and despite the radar showing precip. we just can't get anything heavy enough to stick!! Yet, just 7 miles away, the roads are covered. Beyond frustrating...

    ReplyDelete
  11. Alex is right about the forecast--for the Kitsap Peninsula and all the areas close to Puget Sound the forecast for last night (Tuesday night) should have read "windy with north wind 20-30 mph with gusts to 40 mph".

    The forecast is wrong sometimes. There are a handful of good reasons for this and a few excuses that may or may not add up. I could do a blog on it, heh.

    ReplyDelete
  12. Less whining, more predicting!! Abandon the nws if it so bad. Use Europe's! In fact lets give money to Europe to further develop their supperior system and maybe we will get something phenominal.

    ReplyDelete
  13. Banjo - Oh boo hoo! Supporting science shouldn't be political, but nowadays it is. This is Dr. Mass' blog and he will write about what he wants. But stick around and keep reading. You might find that what you see as "political" is actually common sense.

    ReplyDelete
  14. Alex Taub (and all modelling cynics) -

    You may have a point about your dog, the radar and your wet finger for the immediate now, perhaps even an hour from now as well as your little tiny locale where your own eyes have some insight, but that is hardly a good test of modelling.

    Are you aware of chaos theory under which mathematically ( very high predictability) it is calculated that beyond 9 days ahead, weather forecasting is impossible, even if your models have perfect predictability function? Weather is a chaotic system, like so many others.

    Now consider this: Modern weather forecasting has attained ( I don't know the stats, but bet Cliff does) remarkably functional reliability in accuracy of forecasts for up to 7 days ahead. That is, we the user can rely on them to at least start planning, with some confidence that the forecast will represent a good approximation of reality. That means we can better manage risk, far better than any wet finger or advice from your dog as to wether it will be kinda sorta maybe windy in an hour right here but spin a bottle for across the bridge.

    Now consider this: this remarkable closing on the absolute mathematical limits has been almost exclusively a result of ever well resourced and refined computer models. come to think of it, even your car is more reliable due to modelling. Perfectly reliable? of course not. More reliable than your old Edsel? you better believe it, no matter how nostalgic you are.

    ReplyDelete
  15. Cliff,

    Another great post. Call me a cynic, but I don't expect any funding to the NWS/NWP from the GOP congress or GOP President. In my experience, they have shown that they're more interested in sending those government tax dollars to private corporations instead, who aren't accountable to citizens, only profits instead.

    Also, I love that you don't shy away from politics. It is intertwined into everything, including U.S. weather forecasting. It can't be ignored and I'm impressed that you don't. Thanks for a great post.

    ReplyDelete
  16. Cliff, what's up with all this dry weather? Seems like all of our precipitation is dien in California?

    ReplyDelete
  17. I don't think the problem is computing. I follow, as can any of you, the models used by NOAA, by TWC, WU, etc, on a site called pivotalweather.com. If you look at this site, it will show you just a taste of the variability of atmospheric conditions that forecasters have to deal with.

    I don't think computing power is the problem, the prediction seems to suck at the NWS. But if you look at recent storms, and check multiple sources, as I do, you can see many of the outlets screwed up badly with the last round of storms. Places that were not supposed to get tons of snow did. Places that were supposed to get tons of snow didn't, like far eastern portions of Washington State. I mean, how can anybody forecast that the latest storm, that strong low off Portland, would just sit there all day yesterday and into this morning before dissipating?

    Great post, but I believe the problem is just bigger than computers. I believe the weather is getting more difficult to predict, because of changing atmospheric conditions (sorry, read climate change). And the extremes are getting worse, the pendulum swings of weather. Just look up what happened lately in the entire West Coast lately to see extreme weather at work.

    ReplyDelete
  18. Agree with Moral Individual. As with medicine, the 21st century scientist often thinks we are close to knowledge of incredibly complex and chaotic systems like the brain or global weather. Better to just be humble. For Trumps part it should be great. He will most likely drain the swamp of politicized science and restore the alphabet soup of bureaucracies to their more humble and intended missions.

    ReplyDelete
  19. You do realize don't you that the average swamp has many beneficial organisms in it, even if you dislike certain leaches?

    In another solar system, in another galaxy, there is another web forum that discusses exactly the same notions of "error" by exactly the same sort of political animal, but this time regarding the forecast flood levels of a particular river in California. The forecasts were the dreaded "models" of course and they plotted a familiar pattern of uncertainty and adjustment as feedback informed theory as the flooding progressed.

    Always, and I do mean always, the "conservative types" were highly suspicious of reliability and they gleefully jumped all over what they termed "errors" or "flaws" in how the forecasts did not perfectly match the reality (only seen in the brilliance of hindsight).

    This is predictable, more predictable than weather. Authoritarian personalities are measurably more hostile to ambiguity and risk. That is, they compartmentalize their "world view" as black and white, good and evil, up and down and no shades of grey. As well they are risk averse and counterintuitively, that is not risk competent - quite the contrary.

    Knowing this, the hostility to, cynicism and denial of modelling as a credible means to anticipate future events is to be expected by those who identify as conservative in politics. They value and in fact demand certainty. When we explain that any and all forecasting involves uncertainty one way or another, this triggers their defences. It is battle stations from there on.

    http://www.smithsonianmag.com/science-nature/study-predicts-political-beliefs-with-83-percent-accuracy-17536124/

    ReplyDelete
  20. That the UK Met Office has reasonable accuracy in its forecasts is remarkable, in my opinion. Our relatively small archipelago is very much at the whim of the winds. A small change in direction or intensity of an approaching (or receding, of course) weather system can have dramatic effects on our weather. Another thing I find remarkable is our mild climate, considering London is on the same latitude as the southern end of James Bay, part of Hudson Bay. We had a couple of inches of snow yesterday, and southern England just isn't used to it. Roads closed and massive travel disruption.

    ReplyDelete
  21. My comments are not showing up. Censorship! *Kaboom!* Here that? That's the sound of Trump's big canon coming to clean house! Better get your priorities straight! Do you want a better infrastructure that actually encourages a healthy economy not just quick fixes by higher taxes?

    Then don't vote for people who only care about bathroom modification to please the minority and those who would rather fix murals then fix roads that need it.

    ReplyDelete
  22. Thanks Cliff for an interesting and provocative post. Best of luck on your goals.

    ReplyDelete

Please make sure your comments are civil. Name calling and personal attacks are not appropriate.

Should I Let AI Write My Blogs?

I spend a lot of time thinking about and writing my blogs.   Could AI apps do the work for me?   Could any of you tell the difference? Well,...