December 10, 2012

Rebuilding the National Weather Service

 Weather prediction is a critical national activity.

As noted in various publications, trillions of dollars of U.S. economic activity are sensitive to weather (Dutton 2002), with nearly a half-trillion dollars of variability in our economy due to weather (Lazo et al., 2011).  Furthermore, the U.S. experiences a wide range of severe weather conditions (tornadoes, hurricanes, severe convection, windstorms, snowstorms) that not only results in economic impacts, but the loss of hundreds to thousands of lives a year from intense weather.

The U.S. should have the best weather prediction capability in the world.  We have the world's most extensive weather research community and many key weather prediction breakthroughs have occurred here, including the first numerical weather prediction.  We also spend more money on weather prediction than anyone else and our private sector meteorological community is large and vigorous.

But the sad truth is that we are lagging behind the world leaders, and are even farther behind our inherent capabilities.  The loss to the nation, both in lives and economic value, is immense and unnecessary.   It is time to face up to the problems and fix them.


This year a National Academy of Sciences committee, asked to evaluate the National Weather Service (NWS), completed a report: Weather Service for the Nation:  Becoming Second to None.  The implication of this report is clear:  the National Weather Service is no longer the world leader in weather prediction.

During Hurricane Sandy, much of the media discussed how the best forecasting system for predicting Sandy was not American, rather it was the model of the European Center for Medium Range Weather Forecasting (ECMWF) in Reading, England.  Even USAToday (see graphic) and the NBC Nightly News talked about U.S. inferiority.  When the popular media are talking about U.S. weather prediction inferiority, you know you are in trouble.


But the problems are worse than inferior global weather prediction, which are confirmed by the statistics on the NWS's own web site.  The U.S. is falling behind in the next generation forecasting technology based on ensembles of many forecasts:  probabilistic prediction.  Furthermore, the head of the NWS was recently forced to retire when financial irregularities in the NWS budget were found, and mismanagement of the acquisition of a new generation of polar orbiting satellites threatens to produce a gap in critical satellite coverage.  And there is a lot more that I will not mention now.

It is not that the National Weather Service isn't making a huge contribution today--it is.  NWS forecasts during the days before hurricane Sandy were extraordinarily useful, saving property and many lives.  Few government agencies provide such a valuable, useful product.  And the NWS is staffed with dedicated, talented, and well-educated forecasters, researchers, and other employees.  But it is undeniable that this organization is a shadow of what it could bethe dominant weather prediction entity of the planet backed by the largest national research enterprise and the largest private sector weather sector in the world. 

The impact of not being the best is costing the U.S. billions, if not tens of billions of dollars per year, and many lives.

In this blog, I will describe some of the problems and how they might be addressed.  This is going to be a very frank evaluation, perhaps too frank. And let's be clear up front:  the issues are both long-standing and aren't limited to the National Weather Service, they extend into NOAA management and higher.  And without fundamental change, the nation will continue to be served by a critical organization not realizing its potential.

Let me now discuss a few of the problems I see, starting with the most easily addressed first.

Problems 1:  the lack of computer resources.  

The National Weather Service efforts in global and national prediction are being undermined by lack of computer power.  This is not just my opinion--ask leaders inside the weather service, National Academy committees, or knowledgeable folks in the field and they will tell you the same thing.  Why are ECMWF's forecasts so much better than the NWS's?  A big part of it is that ECMWF has the computer power to run their high-resolution global forecast at 16 km resolution, while ours is 25 km, their ensemble forecasting system has twice the resolution, and they can use an advanced data assimilation approach (4DVAR) that requires a lot of computer resources.

Consider the difference in the computer resources available to ECMWF versus the National Weather Service's Environmental Modeling Center (EMC), and keep in mind that EMC has a much wider range of responsibility (includes high resolution national and regional forecasting) than ECMWF (which does only global forecasting).

ECMWF has two machines, each with 24546 cores and a computational ability of .75 petaflops (thousand trillion floating point operations per second).   These machines are #37 and 38 on the worldwide list of top 500 computers.  The National Weather Service has two computers that are not even on the top 500 list.   Each has 4992 processors and an ability to do .07 petaflops.   You see the problem?  The NWS has less than 10% of the computer power as ECMWF and has many more responsibilities.

Ironically, NOAA (of which the NWS is part) and other government agencies have some very big machines (NOAA's Fairmont, .38 petaflop, DOE's Sequoia and Titan-20, 27 petaflops), which are NOT being used for weather prediction but climate prediction and other needs.   Kind of ironic we have huge computer power available for climate prediction, but not for predicting the weather today that critically influences U.S. citizens.  A year ago, at the dedication of the new coastal radar, I was having lunch with the head of the NWS and Senator Maria Cantwell.   The NWS head admitted that U.S. weather prediction no longer led the world and that lack of computer power was an issue.  Senator Cantwell could not believe it, noting how many powerful computers Congress had authorized for national needs.

So why is the NWS stuck with such inferior computers?  It is not like my colleagues at the NWS are not clambering for them.  First, the NWS/NOAA computer acquisition managers have made  poor choices, playing it safe with extremely expensive, "heavy metal" IBM hardware rather than less costly commodity chips (like Intel Nehalem cores).  Second, it is clear that NWS and NOAA management have not given NWS computer acquisition sufficient priority, a clear indication of management failure, since they have known of this situation for years.

Recently, NOAA/NWS signed a TEN-YEAR agreement keeping IBM and its gold-plated hardware as the choice for the NWS (IBM is finally going to use commodity chips, I should point out, but IBM is still very, very expensive).  Unbelievable. Why commit to long-term deal with an expensive vendor?  Experienced computer folks I know were just shaking their heads.

How to fix this problem.  NOAA management has to immediately secure funding for a petaflop-class machine for operational numerical weather prediction (NWP).   Another approach would be to immediately reassign the use of the computer resources already in NOAA (e.g., Fairmont) or other agencies (e.g., DOE) from climate to weather prediction.  We need some rebalancing of priorities.  For the price of one expensive military jet, the nation could have far better weather prediction. 

Bu the problems with the National Weather Service extend far beyond computer power.

Problem 2: Structural deficiencies of how the National Weather Services organizes its research and operations regarding weather modeling.

Case in point: the weather modeling and the research that support it are not in the same organization.

Operational weather prediction is found in the National Weather Service's Environmental Modeling Center (EMC, located in DC), while most of the research and development is location in NOAA's Office of Oceanographic and Atmospheric Research (OAR).   So the director of EMC and the head of the NWS DO NOT HAVE CONTROL OF THE RESEARCH THAT SUPPORTs THEIR OPERATIONS.   Much of the relevant research is being done in Boulder at NOAA's Earth Systems Environmental Lab (ESRL), while other research is completed at the Geophysical Fluid Dynamics Lab (GFDL) in Princeton (hurricane modeling) and additional venues.   The folks in NOAA OAR and the NWS EMC often are not well coordinated, have different ideas on how to proceed, and frequently evince a competitive attitude towards the other.
NWS Organizational Chart
 Even within the NWS there are structural problems.   The key area of postprocessing (statistically improving model output) is not in NWS's  EMC but found in ANOTHER lab (Meteorological Development Lab, MDL) at another location in DC.   The result has been duplication, waste, and lack of coordination. 

Possible Solution:  Concentrate government operational modeling and most Federal modeling research in Boulder, Colorado, by moving EMC to that city.  Boulder is the intellectual center of weather prediction modeling in the U.S.  Not only is the NOAA ESRL lab there, but the center of research modeling--the National Center for Atmospheric Research (NCAR)-- is there as well. Boulder is a much more attractive and central location for attracting visitors and employees than the suburbs of DC.  It makes sense for the Environmental Modeling Center (which includes oceans and atmosphere) to be in NOAA, rather than the NWS.  And a formal connection with NCAR should be made as well, including long-term support of research and technology transfer, as well as adding the NCAR Developmental Testbed Center (DTC).  The statistical postprocessing folks in MDL should join the new organization.  In short, a highly integrated and comprehensive group--encompassing weather prediction operations and research should be established in Boulder, Colorado.  The impact of doing so will be substantial and positive.

Problem 3:  Poor leadership in the National Weather Service and NOAA.

Visionary leadership has been absent from the National Weather Service for a number of years.  For decades it has been the pre-retirement home of colonels, generals and admirals, with a number of them never mastering the complex technologies for which they were responsible.  Just to give you an idea, here are the recent heads of the NWS:

Colonel Elbert Friday, 1988-1997 (USAF, but did have a Ph.D. in the field)
General Jack Kelly, USAF, 1998-2003
General David Johnson, USAF (ex-pilot) 2004-2006
Colonel Jack Hayes, USAF (weather officer with Ph.D) 2007-2012

Importantly, these military leaders brought many military folks into middle management positions in the NWS.  The military leaders of the NWS had a varied meteorological background, ranging from virtually nothing to Ph.Ds; most had a priority of keeping the ship afloat, versus scientific/technical excellence.  They had all managed large organizations before, but clearly many of them did not have a clear vision of how to lead a science-based, technological organization.   During their tenure, the NWS slid from number one in weather prediction to secondary status, with problems like those noted here festering and worsening.

A new, high-level position was created recently in NOAA to oversee NOAA operations (including weather prediction).  You will never guess his background:  an admiral.  And he is promoting a failed bureaucratic approach in which all the weather agencies will somehow work on the same models, called NUOPC.

I am not saying that generals and admirals can't be effective leaders for a highly scientific civilian agency.  But as documented in Thomas Rick's bestselling book, The Generals, our military leadership has become strong on tactics and weak on strategy and vision.  It has certainly been true of their management of the National Weather Service.
I should be clear that NOAA management is also heavily indicted in NWS problems, since the issues really encompass the combined NOAA/NWS entity.  The NWS has suffered from lack of resources from NOAA and the U.S. government.  In fact, the previous head of the NWS Jack Hayes tried to get more resources from NOAA management, but was rebuffed.   As the NWS has slid, few in NOAA leadership seemed to have noticed or cared.  Neither has Congress, until recently.

Solution:  The NWS director position is now open after two failed searches (a story in itself).  This time find a civilian leader with strong scientific credentials and a credible vision of the future of the organization.  If possible, someone with fresh insights from outside of the organization.

Problem 4:  Poor organization of other key National Weather Service components

There are some strong components of the National Weather Service, such as the local office structure that allows forecasters to become intimately knowledgeable about their local weather and to effectively interact with regional agencies, governments, and users.  (I have heard rumors that some in NOAA leadership have proposed to destroy this structure, closing offices into regional forecasting centers.  This would be a disaster).

But large amounts of staff and funds are wasted on the division of the NWS into regions, each with its own leadership and bureaucracy (Eastern, Western, Central, Southern, Alaska, Pacific).  I have a lot of friends in the NWS and most of them believe the region system is outdated and wasteful.

There are also national centers whose functions are redundant with local offices.  A prime example is the NWS Hydrometeorological Prediction Center (HPC), which makes all sorts of precipitation forecasts and prepares old-fashioned surface charts.  Nearly all of their functions are redundant with local offices or could easily be done locally.  My colleagues in local NWS offices believe this as well.

Possible Solution:  End the region structure and integrate all functions into an improved national structure. Reduce or eliminate the HPC and other unnecessary centers.

Problem 5: Lack of rational observing system planning

 Recently, the media has been abuzz with the expected gap in polar-orbiter weather satellite coverage, suggesting weather forecasting skill would decline.   The NOAA/NWS polar orbiter acquisition program has been characterized by mismanagement for years, not only delaying the next generation satellites, but costing the nation billions of dollars.  Even Jane Lubchenco, head of NOAA, acknowledged the polar orbiter satellite acquisition had become a “national embarrassment.”  But the problems are deeper than satellite development and acquisition.
JPSS Polar Orbiter Satellite
Numerical weather prediction is based on using a wide range of observations, from billion dollar weather satellites, to surface observations, radiosondes, aircraft observations, and other collection platforms.  The NWS needs to assess the optimal collection of observations, including dropping old data sources that are no longer needed.  There is a rational way to do so through Observing System Simulation Experiments (OSSEs) and Observing System Experiments (OSEs).  In the first, we simulate (forecast) for an extended period and make believe that is reality.  Then we sample from these simulations, taking pseudo observations, and then simulate the impacts of varying observing systems and strategies.  In the OSE approach we run forecasts and drop varying observing systems  to see the impacts.  To decide whether we need a new satellite system, you need to do extensive OSSEs.  Unfortunately, the NOAA/NWS has done very little of this and billions of dollars of hardware is being purchased without a clear understanding of the impacts of new sensors and satellites.  (My discussions with high-level folks in NOAA confirm this, by the way).  There is a lot being said about major negative impacts of losing one of two polar orbiting satellites without enough information to back it up.  Congress needs to insist on this.

Problem 6:  Lack of interactions/cooperation with the meteorological community

The National Weather Service is a major component of the weather prediction enterprise, but only one part.   A large academic research community produces scientific and technological advances that the NWS needs to support and take advantage of.   A large and healthy U.S. meteorological private sector uses NWS model output and observations to provide a wide range of weather products for the nation.  Thus, the NWS cannot act in a vacuum and needs close interactions with the entire community.  Decisions on everything from modeling to data formats need close cooperation between the NWS and the rest of the community.

Unfortunately,  this has not been the case.  The National Weather Service, and particularly its weather prediction entity, EMC, have had a tendency towards isolation, lack of openness to other approaches ("not invented here attitude"), and a habit of putting out plans without much community input.  The fruits of nation's research community are not flowing in the National Weather Service.  In fact, foreign centers (such as ECMWF) have been the eager recipients of U.S. research and development advances, when NWS has shown lack of interest.    A good example of the problem occurred in the late 90s when  a new modeling system (WRF, Weather Research and Forecasting Model) was developed that was going to bring the academic and research communities together, to the great benefit of both.  The NWS then decided it wanted to use its own model, one that was inferior by objective standards.

Another example: National Academy committees and the private sector have recommended and pleaded over the past decade for the NWS set up a formal advisory committee to provide community input; such an advisory board still does not exist.  Recently, Tom Bogdan, head of the University Corporation for Atmospheric Research, is working to set up an alternative advisory group:  the Weather Commission.  Perhaps that might be a fruitful approach to the problem.

You would expect that the NWS would provide substantial grant support to universities working on research with applications to NWS goals. You would be wrong.  Funding for extramural research has been miniscule, and such research is the first thing to be cut when NWS budgets get tight.  Believe me, I know about this intimately.

The insularity of NWS personnel has recently been substantially enhanced by the cancellation of nearly all travel to important meteorological conferences and workshops.  A penny wise, pound foolish policy at best.

Finally, documents describing NWS future plans are generally vague and uninformative, with heavy reliance on catchy, essentially meaningless, phrases such as "weather-ready nation."  (It is amusing to listen to how many times administrators and managers like to mention that line).  If you want to see what I mean, check out their latest "strategic plan."   Where is the specific plan for becoming the number one weather prediction entity in the world, with adequate computers and staff?  That plan you won't find anywhere.

.....................

This blog is getting long, and there is a long list of NWS issues that I haven't gotten to, including profound problems at the National Climatic Data Center, the national repository of weather and climate information (it is extremely hard to get relevant weather/climate data, particularly for someone without an intimate knowledge of NWS datasets).  Or the ill-fated attempt to create a parallel National Climate Service.   And when I talk to middle-level managers in the NWS they complain about powerful unions that slow down innovation and new ways of doing business.

The National Weather Service and NOAA are simply not providing the nation with the quality weather forecasts and weather information they should, and a substantial amount of public funds are not being used effectively.  Which is a terrible shame considering the need, the extraordinary talent within both NOAA and the NWS, and the nation's scientific dominance in the field.

Better, more visionary, management is needed.  It is become more than clear that current NOAA/NWS management are incapable of fixing the problems themselves.  Congress, with its oversight responsibilities, must intervene and the meteorological community must weigh in on needed changes.

This blog is going to get me in trouble with some, I know that, but the stakes are too high and the importance to the nation is too great to keep silent.

Comparison between the U.S. (GFS, black), European Center (red lines), and other modeling systems over the northern hemisphere.  The closer to one the better.  The ECMWF model (red) beats the U.S. model (black) for nearly all days.  The difference is even larger in the southern hemisphere.

30 comments:

  1. Rather disturbing middle-of-the-night reading. It sounds like this situation has been brewing for a very long time. What has happened to keep you up this late in deciding to bring it to public attention now?

    ReplyDelete
  2. Cliff, yes it was a long blog and I read every word. I hope it is read far and wide.

    I, however, am not optimistic about the prospects for the future of meteorological services in the United States. The numerous problems you summarize in this post are a symptom of what I believe is a far reaching underlying problem, that we now have a "government class" to go along with poor, middle class and rich.

    I am not saying there is a conspiracy afoot. But to me it just seems very clear that government now consists of people more intent on preservation of a status quo that serves themselves rather than being passionately focused on serving the public. That's where things like a 10 year deal with IBM for inferior hardware come from.

    I think our best hope is to dissolve the government class and rebuild from first principles of our government and public service.

    We must constitute our government with citizens with not only the highest intellectual capability, but also with the highest moral character (that comment ought get some people riled up).

    Do I think the NWS can be rebuilt to competitively serve the people of the United States with excellence? Yes, but not in the vacuum of an entire government going the wrong way.

    ZCZCZC END RANT

    Best Regards

    ReplyDelete
  3. The NOAA management issue is more severe than you state. NOAA senior management officials such as the former CFO who is now the Chief or Resources and Operations and others have severely cut NWS proposals for more research funding. In addtion, they are the ones who forced EMC down the path of IBM by useless "oversight" and inane review boards by the number!
    Dr Hayes tried hard to influence change at NOAA to no avail.
    Morale here in the NWS Headquarters is the lowest I have seen it in my many years.

    ReplyDelete
  4. Cliff - Your article is interesting, but leaves out a glaring problem...the cause of the budget issues. NWS is funded at a level of roughly $1B per year (not NOAA, which is much larger primarily due to satellites). That should be more than enough to provide world class meteorological services, yet their are those within the organization who believe the only answer is "more money"

    While you suggest more of a national structure and eliminate or reduce the regional structure, which I agree with, the fact is today the organizational structure is antiquated and has not kept pace with technological change and the productivity it offers. 80-85% of the NWS budget is taken up by manpower, which supports the antiquated structure. A heavily antagonistic union, probably one of the most aggressive ones in the Federal Government, has been permitted to fight any action designed to streamline the agency...and the professional graves of the last two directors are littered with union influence.

    Also, Congress, in its infinite wisdom, has granted in past years (not the last two) pay raises...authorizing the raises but never appropriating the funds for those increases, challenging the agencies to take that out of hide. With an 85% manpower rate, that leaves 15% to eat those raises. Consequently, the needed technology refresh and sustainment funds are raided to pay the bills, and performance suffers.

    Quite simply, NWS doesn't need 122 forecast offices. Performance could be improved, money could go back to research/technology refresh, with many less offices, but the union and the local Congressman, who is willing to fight for "his office" is the root cause. Pretty simple fix. Last two directors tried to take it on with the "Conops" discussion, but the union took them on and found sympathetic ears on the Hill, which killed the innovation.

    The organization is in a death spiral. Soon, and because of their actions, NWSEO will stand proudly besides the Hostess Twinkies union. Tactical after tactical victory...but lacking any strategic vision of their own other than to articulate the need for more money, NWS will die.

    Such a shame.

    ReplyDelete
  5. The situation you describe is another iteration of the government is too big to succeed. Private weather forecasters seem to be the most accurate in the US. Why not privatize the NWS?

    I know, wishful thinking. Nothing will be privatized during the next 4 years. We will continue to squander money upon the dysfunctional US Postal Service, AMTRAK and the like.

    ReplyDelete
  6. NOAA's and the NWS problems are systemic. NOAA does not have an organic act that ordains that the organization is overseen by a single committee in each house of Congress. Instead, each of five line organization reports to a different committee. NOAA's budget goes through the Commerce Dept. and the OMB on its way to Congress, where it's heard in the Commerce committees. Priorities and strategies are modified at every step. Any major reorganization requires an act of Congress.

    The only solution to the weather prediction/computer problem is through a specific Congressional line item that has the support of a Senator or two. Assuming it isn't waylaid by the bean counters who get their weather forecasts from TV stations.

    ReplyDelete
  7. As anyone in IT will tell you, buying hardware from IBM is almost always a sign of a CYA attitude at work. The saying used to be, "no one ever got fired for buying IBM."

    Paddy: If privatizing the NWS were a reasonably idea, you'd see the private weather forecasters clamoring for it. The reason they're not is they all rely on the NWS's observation network. The only pieces they want to privatize are the end results -- the forecasts. (This is why they fought for a while against letting the NWS provide grid forecast data, several years back.)

    It does sound like we might want to consider devolving the NWS into just collecting observational data, and then letting the Europeans run the models, since they seem to do it best.

    ReplyDelete
  8. I cannot tell you how much the employees of the NWS appreciate your comments, I assure you, your post is being widely distributed across the agency. You are spot on in your analysis of the current state of the NWS and of NOAA and I agree to a 95% level as to the cause of our problems. Where I disagree, is largely the underestimation of NOAA and their coveting of NWS resources as the main source of the problem. In spite of the issues outlined in your post, I see a very bright future for the National Weather Service, obviously after we make it through the doldrums we are currently in. Just for example, no organization anywhere in the world is in a better position to further our knowledge of operational mesoscale meteorology than the NWS. If we could only get NOAA to allocate some of it’s over $700 million a year in grant money to support meteorological research; re-wire the management of the agency to take advantage of its innovative employees at 122 locations across the country and a small investment in mesoscale modeling; mesoscale meteorological knowledge and forecasting techniques would expand exponentially. If you think about it, isn’t it silly that in 2012, we still have forecasters sitting in front of radars issuing tornado warnings, much the same as I did as an intern in the early 1990s and others before me did in the 1970s. Surely by now we should be modeling radar returns getting a jump on warnings. This expansion of knowledge would not only save lives it would have a direct economic impact on both private sector weather companies and out nation’s economy as a whole.
    Mesoscale meteorology isn’t the only opportunity before us, space weather, water resources, climate research and operations to just name a few bring fantastic opportunities. We will get there, I am sure, with the help of forward thinking people like you and the folks at the Weather Commission as well as the many others who care about this agency, we will turn this ship around. We were very close just two years but as Concerned (who just hit it out of the park) pointed out above some leftovers in NOAA and the NWS itself were able slow the ship again by getting rid of Jack and Bob. Thanks again for your excellent analysis and thoughtful comments.

    ReplyDelete
  9. I'm afraid that this one has gotten away from us, and it may be too far gone to fix it. If we can fix it, this will take a lot of willpower, because there is going to have to be a fundamental change in how the agency operates. Historically, the U.S. government has not done well in such endeavors - it's solution is usually to just create yet another layer of bureaucracy.

    A big monolithic agency is no longer warranted. Instead, like with the military, we need the equivalent of a rapid strike force to handle things like hurricanes, plus a top-notch research and modelling group to support the strike force. To do that, a lot of heads need to roll, and I don't think anyone has the guts to do that. Heck, just moving an office turns into a political pissing match of epic proportions, with senators, congresspersons and local politicians all trying to keep the office where it is.

    The fact that military desk jockeys are running the show is not the cause of the problem - it is the symptom. There are many agencies where the military approach to things make a lot of sense (e.g., nuclear energy), but weather prediction and research is not one of them!

    ReplyDelete
  10. Oh, but Sysiphus, the big thing in not really just about Hurricane forecasting. It is about the day to day, 365 days a year forecasting. So the rapid 'strike force' idea won't work.

    ReplyDelete
  11. I like how Dan didn't disagree with the part about the union slowing down innovation. And the part about Mesoscale models forecasting thunderstorms.

    ReplyDelete
  12. @Dan As pointed out in Cliff's post, NWSEO is part of the problem. You said nothing at all to address that. While NWSEO has taken up good fights (ITOs for example), by and large NWSEO exists to slow progress, stifle innovation, and allow under-qualified and, dare I say, lazy meteorologists to persist in local offices.

    At the local level, NWSEO representation can be highly beneficial, allowing all NWS employees to work together toward a common goal of continually enhancing services to our customers. It can go south quickly though, where large disagreements erupt of simple placement of monitors (change in working conditions!). Attempts to alter or enhance our services can be met with strict denial as well at local and regional levels, with an undue large burden on managers to show beyond 100% the value of trying something different. Once you get to the national level, you have a group of entrenched representation that does not speak for all NWS meteorologists, especially the younger more aggressive crowd who instantly realize the value in becoming involved with social media, providing video briefings, and getting out of THE ZONES mentality. Rather NWSEO is seen as sticking up for those that insist on as slow and little change as possible, seeing nothing as broke (so why fix it?), and completely oblivious to the logistical constraints senior leadership must work with.

    NWSEO also refuses to even consider the possibility that the current structure of the NWS should change at all. Apparently the position of NWSEO is that having 122 field offices with a staff of 25 at each office is "just right"; that having one office serve the 12M+ people in the New York metropolitan area while five (5!) offices with a combined staff of over 100 serve the roughly 1M people of North and South Dakota is "just right"; that it is more important that the all important ZFP/text products are "just right", ignoring that we are completely a digital society now with much higher temporal and spatial needs that are not served well at all with a text product of gross resolution from several decades ago. NWSEO should strongly reconsider this position with "service above self" in mind; that a small group of local NWS offices can be absolutely crushed by active weather (hurricane, tornadoes, blizzard) while literally hundreds of meteorologists sit idle across the rest of the country is "just right".

    Yes, the "higher ups" shoulder a lot of blame for the falter of NWS, especially when it comes to modeling efforts. However they are not alone. How can an agency move forward when the single entity for representing employee's voices is so out-of-touch with the real needs and resources?

    ReplyDelete
  13. orv, the no one ever got fired for buying IBM meme went out the door long back. Now IBM has vigorous competition in practically every field in which it competes, and from personal experience, we quite often pick non IBM products.

    ReplyDelete
  14. Cliff -
    Very insightful. I'll share your article.

    Chris Tomer
    FOX-31/Channel 2
    Denver, CO

    ReplyDelete
  15. How about taking a page from the SETI@home people, in which spare CPU cycles on home computers are used to analyze radio-telescope data for potential extraterrestrial intelligence? Could this type of approach be applied to US weather forecasting, distributing the job out to thousands or millions of computers?
    Ideally of course the NWS should be funded to have top-notch computing facilities, but since that may not happen anytime soon...

    ReplyDelete
  16. "Oh, but Sysiphus, the big thing in not really just about Hurricane forecasting. It is about the day to day, 365 days a year forecasting. So the rapid 'strike force' idea won't work."

    Fair enough. My comment was half tongue in cheek. The point is that for day-to-day stuff, we of course need good meteorologists, but the real lack of resources there at the moment is with the models and with how the agency is being run. Resources are not infinite, particularly with the deficit being so high, so we have to spend the money smartly, where it will do the most good. In terms of leadership, the NHC has had its ups and downs, but I think most would agree it has been doing a really good job lately. Perhaps that is a model? I don't claim to be an expert or have any deep knowledge of the situation, unlike Cliff and some of the posters here.

    ReplyDelete
  17. Paddy L,

    Where would a privatized weather service get its raw data? *crickets*

    I don't think that money WISELY spent on services that benefit us all, like the Postal Service, AMTRAK, and the NWS, is "squandered"... tell me, do you maintain the road in front of your house yourself? Me either.

    There are many changes that CAN and SHOULD happen in the weather service, and I'm glad, as always, that Cliff is sharing this info so we can take action.

    Dismantling and selling off a good but imperfect system is just stupid.

    ReplyDelete
  18. "How about taking a page from the SETI@home people, in which spare CPU cycles on home computers are used to analyze radio-telescope data for potential extraterrestrial intelligence?"

    Brian, Not possible as Numerical Weather Prediction requires very high speed/ low latency communications between cores/computers as the atmosphere is broken up into pieces and run on different cores and nodes. The pieces than need to talk to each other to exchange information and have to wait until that exchange takes place before the model can go on, which occurs many times a second. Even Gigabit Ethernet isn't fast enough. Most large machines used for NWP use Infiniband which is super low latency (1 to 2 microseconds) and 32 Gbit/s or higher.

    ReplyDelete
  19. I did not address the thought that the NWSEO is slowing down innovation because there is no truth to it. In fact quite the opposite is true. Sure there have been times when the negotiation process slows down the pace that ideas move forward but most of the time the reason for this is that management holds onto a proposal until the last minute or we provide counter proposals and management never respond. It is true that sometimes negotiations slow the pace of change because the union does not like the change but the reality is that happens rarely. The fact of the matter is the concept of weather ready nation, not the name, the concept started as an NWSEO idea. The NWS strategic plan was built from the ground up through the NWSEO/NWS LOT process. The NWSEO was instrumental in the rapid spread and acceptance of NWSChat. We have worked with the NWS to develop a whole new way of doing business with our hydrology program; we now wait for the bureaucracy to implement it. The NWSEO has pushed the NWS for years to be more involved in social media and to develop weather apps for phones and tablets. We run into resistance from management who argue that they must “control innovation in the field”. At one Corporate Board meeting about two years ago one regional director complained at length that one of their employees was on the phone to Google for three hours trying to learn about innovations in communications. I finally could not take it anymore, interrupted and asked if they thought managers at Google were complaining about one of their employees talking to someone in the NWS for three hours.

    cont.

    ReplyDelete


  20. As far as 122 being the optimal number of WFOs, it absolutely is not. I think we need WFOs in Charlotte, Baltimore and Ft Myers/Naples. As far as what is the optimal staffing we are open to ideas. The NWSEO proposed and fought hard to get pilot offices in six offices (these pilots were 100 percent a NWSEO idea), the whole idea behind these offices is to innovate for the future. It was a very hard fight, management was very resistant to the idea, but the foresight of folks like Jack Hayes and Bob Byrd finally helped and they agreed to fund these most of the pilots. It is almost the one year anniversary of the NWSEO proposing to the NWS to move some intern positions out of offices that do not have upper air and have a small population base to locations in the Midwest to create mesoscale/tornado pilots. The idea was for the NWS to announce the new pilots on the anniversary of the Joplin Tornado. The NWSEO thought that would be a very powerful media message and the NWS would show they could respond rapidly to a need. The NWS management would not agree to do this even after complaining that the NWSEO would not agree to anything but cookie cutter staffing. It has been the position of the NWSEO for as long as I have been involved that we probably do not need a WCM in every office in say the desert southwest or in sparsely populated states. As for the SOOs, while it is a very important position, the current management of the position has led to much of the dysfunction the Cliff speaks of in his post. NWS OST needs to be changed to more of a direct R&D program for operations, Dry side NOAA labs (or at least portions of them) should be brought under this new OST management; SOOs should then be matrix managed between this new OST and the local/regional office. This would create a direct connection between research and operations and well as cultivate the tremendous talent in local offices. I personally have been proposing this idea for four years, the NWS management hates it.

    As the President of a labor union my number one job is to save other peoples jobs, why on earth would I or the organization I represent not encourage innovation that would open new markets for the information the NWS produces. It just makes no sense. However don’t come to us with some plan to cut hundreds of meteorologist’s positions, because NOAA will just take those positions and transfer them to the wet side or to NOAA administration and we will never get them back.

    ReplyDelete
  21. Also, since weather forecasting is time critical, I presume you wouldn't want to do it with systems that aren't 24x7 (not just SETI style or even cloud based systems with redundancy). I exclude non critical jobs (presumably there are research models or past analysis jobs and the like) that could be run on other systems.

    I am not a meteorologist, but I am a computer person, and I don't see anything wrong at all with the use of an IBM supercomputer (I don't work for IBM or the NWS). IBM is probably the world's leading manufacturer of supercomputers and has a great deal of expertise in this very complex technology. Plus, its not just raw power, but also 24x7 support, software upgrades etc.

    ReplyDelete
  22. @Dan sounds like under your leadership NWSEO likes to take a scattershot approach to problem solving, hope something sticks, claim a win when it does and blame the other guy when it doesn't. The fact is NWS has stagnated while you have been actively involved in the upper levels of NWSEO. You continue to support a WFO structure that does not leverage technical advances of the past 20 years. NWSEO regularly fights for the local forecaster improving the forecast (see your rant about mesoscale meteorology before) when objective data clearly show forecast models routinely beating NWS forecasts. Why not support moving regular forecasting to regional or even national centers? WFO collaboration remains a significant issue and hindrance to any customers using NWS forecasts at a national scale (see NY Time graphic depicting NDFD winds with Sandy). The grievance process makes it very difficult if not impossible for NWS to get the very best meteorologists, rather once someone is in they are set forever which leads to complacency and a dulling of skills over time. I agree job safety is important, I disagree that people should be so comfortable that they feel they don't need to learn new skills. Just look at the adoption of social media, which you tout as a success.

    NWSEO needs new leadership, committed to providing Americans with the best scientists with the best skills to safeguard them from the harsh realities of Mother Nature instead of fighting to keep jobs for the sake of keeping jobs.

    ReplyDelete
  23. Something I didn't see expressed... why not join forces with the Europeans (and others) and work together to build the best model and share resources? Why does forecasting have to be a national resource?

    I know the military types will say its for national defense, but in today's world, that's lame. We are all interconnected to a far greater degree than nationalists like to consider.

    One world, one weather forecasting system. EarthWeather.

    John Marshall

    ReplyDelete
  24. John,
    The reason the U.S. needs to be first rank for a number of reasons. First, the ECMWF model is a global one, and we need high resolution forecasts over the U.S. that they will not be doing. Secondly, the global model drives the regional and local models, and we need to get that output at full temporal and spatial resolution...much easier to do if you have it in house.
    Keep in mind that the NWS forecasts for Sandy...the ones made by local offices and the National Hurricane Center, DID take advantage of the European Model. In fact, U.S. forecasters view the output of many global models (e.g., the Canadian, UKMET Office, etc).
    Finally, the U.S. model could be the best in the world...why settle for the ECMWF model when we could be far better?...cliff

    ReplyDelete
  25. I applaud the recommendations for rebuilding the NWS – e.g., the provision of needed computing resources, co-location of organizational components like the EMC with ESRL in Boulder, hiring as new head of NWS an individual with strong scientific credentials and a credible vision of the future, and elimination of the regional structure. They can help rebuild the NWS.

    But if we are concerned about why ECMWF forecasts so much better than the NWS, it is instructive to look at the nature of the ECMWF organization.

    • Accountability – The Director of ECMWF reports directly to the heads of member Met Services in Europe; they have a vested interest in his performance, and he is accountable to them and serves at their pleasure.
    • Mission – ECMWF is able to focus on making and improving medium-range global forecasts, leaving short-term, regional/local forecasting and in-situ/satellite observing systems to others.
    • Pay Differential – The pay for ECMWF professional staff is significantly greater than their peers elsewhere, thus helping attract the best and brightest.
    • Personnel – The Director of ECMWF has significant flexibility in hiring new and firing non-performing professional staff.
    • Transition from Research to Operations – The professional staff throughout ECMWF ranges from basic researchers to operational forecasters, thus enabling the transition from research to operations to take place at the most basic level within and across the organization.

    While the recommendations such as noted above will be necessary to rebuild the NWS, I submit that they will not be sufficient for the NWS to ever become the dominant weather prediction entity of the planet, at least in medium-range forecasting. Additional organizational factors – such as are in play at the ECMWF – must be addressed.

    ReplyDelete
  26. A well-stated and though-provoking piece. There is one issue, however, where I disagree: the idea of dismantling the HPC due to redundancy. I use HPC products not only as a meteorologist in the federal government but also as a weather-hobbyist, and I think they provide a much-needed, national-level assessment of current and forecasted weather. I can recall numerous times when the gridded concatenated products from the various WFOs were marred by greatly differing opinions from field office to field office. The HPC provides a single-source assessment and forecast for those of us who need such a product. As one of the authors of the US Drought Monitor, I rely on the regionally consistent HPC products for the "Outlooks" section; we do not have time to go to different WFOs to cobble together an outlook ourselves for the contiguous U.S. Their QPF products are used in weekly briefings to senior-level staff here at USDA. The HPC (having worked next to them during my time at NESDIS) also offers some of the most knowledgeable forecasters in the business. In additioin, they are a much-needed source of logistical information, such as model initialization biases and data ingest problems. I'll add that many of my co-workers and weather-hobbyist friends find the HPC short- and medium-range discussions quite useful. Anyhow, I know this was not a key piece to your article, but I do feel the HPC serves a need for meteorologists in both government and the private sector, and I also know of many weather-hobbyist friends who use HPC products as well.

    ReplyDelete
  27. link to original report is broken. this one works for me:

    http://www.nap.edu/openbook.php?record_id=13429&page=1

    ReplyDelete
  28. 1 option I see from this and your previous posts is the opportunity to leapfrog our "competitors" with a major re-write of the WRF (or NIM) models with massively parallel GPU architecture in mind.

    Relatively low cost GPU type processors can better handle the 4dvar assimulation tasks.

    With massive parallel hardware running a global model feeding relatively low cost GPU computers running downscaled versions for regional, ensamble and specialty purposes, nearly desktop level GPU processors can provide a lot of specialized modeling services.

    In this case, the National Govt. role is to provide the infrastructure (satelites, radar data assimulation, global models, etc) as well as the data sharing and common software and then providing those services to a wide range of subsequent providers.

    Downscaling, localization of model parameters and increasing the frequency and timesteps can be based on local weather/safety/commerce needs.

    ReplyDelete
  29. Just completed 42 years of government service in Meteorology, and nearly 20 years in Hydrology and recently retired from NOAA/NWS.

    Have seen it all and done most of it including IT and software maintenance (Navy Centers, RFC, and local WFO office).

    Many very smart people (highly regarded) continue to feel the problem can be solved by automation. I only wish that were so. It simply does not scale to staffing requirements at 122 WFOs.

    My Experience: More computers -> More software Maintenance -> More Localization -> Need for more People to Service, Run and QC more Products -> More Training -> Larger Budget Needs.
    i.e. More begets much more.

    Computers enable the production of a greater number of products, supported by more operational and maintenance staff working under a much greater workload.

    For example:
    At the WFO, a conflagration of software from a variety of different agencies must be cobbled to work in an unorchestrated systems environment. (I know. I had to deal with it all the time.)

    One size fits all, yearly major software upgrades and changes upend local development work and require extensive retraining by staff and local modifications.
    Exponentially increasing number of products (created at the National level), requires human attention and interface at the local WFO level and more training.

    The drive to fine scale local detailed forecasting is exponentially increasing mapping/GIS requirements in both Meteorology and Hydrology without dedicated GIS support at the WFO level. Work that is done, is out of hide by multi-tasked forecasters, who have done admirable if not heroic work.

    The absolute dedication of our younger folks, who continue to expend themselves making this Rube Goldberg system work is amazing. But with staff cut backs, it is just a matter of time before the whole thing implodes and people bail.

    In a nutshell: The NWS is trying to do too much. Much like an over-expanded business. The extra support products have been done as a means to impress Congress, justify funding requests and keep jobs.

    The last thing anyone wants, is for dedicated NWS employees to lose their job. However, the NWS needs to reevaluate what it is attempting to do, and refocus on what it can do well under the new budget constraints.

    This may mean a refocus towards impactful weather warnings on very short time scales with a better staffing plan at a smaller number of WFOs using the reduced staffing level that currently exists. The rest should be left to private industry and/or National/Regional centers.

    ReplyDelete
  30. very nice post
    two thumb up for you ^___^

    ReplyDelete

Please make sure your comments are civil. Name calling and personal attacks are not appropriate.