U.S. Numerical Weather Prediction is Crippled by the Division between NOAA and the Academic Community. But a Rare Opportunity Beckons.

U.S. global weather prediction run by NOAA/National Weather Service is now in fourth place among national centers and FAR behind what one would expect from the world-leading U.S. weather research community.

Why?

The answer is clear: the vast U.S. weather community does not work together effectively in developing weather prediction models and transitioning research to operations.

NOAA has highly competent  and motivated weather modeling researchers in its several labs and in the National Weather Service. The center of U.S. academic research in meteorology and modeling is located at the National Center for Atmospheric Research (NCAR) in Boulder, Colorado, an entity run by consortium of U.S. universities (UCAR:  the University Corporation for Atmospheric Research).   The U.S. Navy also does research and development on weather prediction models in places like the Naval Research Lab in Monterey.   But with all these labs and researchers, the U.S. cannot field a world-leading weather prediction model.

Here is the issue:

• NOAA/NWS has its own models and is developing its own global prediction system called UFS (Unified Forecast System).
• UCAR/NCAR has its own models and is developing its own global prediction system called SIMA (System for Integrated Modeling of the Atmosphere).
• The military (mainly the U.S. Navy) has its own models and developing its own global prediction system.

There are all trying to do the same thing.  Spending many tens of millions of dollars. 

None are state of the art.

The separate development of the same type of global prediction model has been a disaster for the U.S.:

• NOAA/NWS lacks the innovation, manpower, and ideas of the academic community.
• The academic community work lacks sufficient resources and misses the effective transfer of its research to societal needs.
• The Navy modeling system is lagging behind NOAA in forecast skill, with obvious implications for national security.

Thus, both government and academic efforts fall short of being state of the art in global weather modeling and prediction.  Progress is slowed.   Tens to hundreds of millions of dollars are used poorly. The costs to society are huge.  You care about global warming? State-of-science numerical weather prediction is a first line of defense against severe weather.

Turf Battles, Ego, and the Inefficiency of Big Bureaucracies

How do I put this discreetly and without offending anyone?

Big governmental bureaucracies are not known for efficiency and innovation.  Surely, I know of exceptions to this statement, but from what I have seen, folks get comfortable in safe government jobs.  Jobs in which excellence is not heavily rewarded and where failure in a Civil Service position does not risk job loss. Government bureaucracies tend to expand over time, with little trimming of inefficient branches or activities.  In NOAA model development and responsibility for model development are split over several, often competing, offices.  And frequently they don't want to help each other or work together, attempting to protect their piece of the pie and turf.  Let me emphasize that I have great respect for NOAA researchers--they are working in a broken system.

And scientists in major government-supported labs (like NCAR) can also become content to play in their own scientific sandbox, seeing little reason to work with others.

Then there are the human frailties of scientists and scientific administrators.  Too many times I have heard NCAR weather modelers dissing NOAA folks, and sometimes the other way around. At a meeting to discuss working together on a major joint effort (the EPIC modeling center), an important university lab administrator said disparaging remarks about his counterpart in the National Weather Service. Such toxic interactions make cooperation more difficult.

Why Isn't Competition Working?

When I write blogs on this topic, inevitably someone brings up the point of competition?  Isn't it good to have many U.S. groups trying to do the same thing?   My answer:

• We have a lot of competition going on right now in the U.S. weather modeling community, and it doesn't appear to lead to a superior approach.
• Global weather modeling is one of the most complex tasks our species attempts: simulating the future from the molecular to the global scales, requiring complex data assimilation, model development, and statistical post-processing.  Weather models (often coupled to ocean, ice, wave and land-surface models) encompass hundreds of thousands to millions of lines of code.   They require the largest computers on the planet.  A HUGE effort.  We didn't have three Apollo programs in the 1960s.  Instead the complex problem was divided among groups around the nation.   But  in weather prediction we have three groups attempted to do something even more complex.
• There are certain key problems that need to be solved by everyone, such as in model physics.  Only by dividing up the problem can we attend to all the issues.  Having three or more teams trying to do the same tasks, inevitably leads to important work not being done well or at all.
• Money and science/technology talent are limited.
• There is plenty of international competition, with other groups working on global prediction (like the European Center)

Congress Knows There is a Problem.  They Proposed a Way Forward.

Problems with U.S. global weather prediction are well known and has been covered extensively in the media (e.g. here) and in this blog..  Congress has had several hearings on the subject (one in which I testified last year) and they have passed legislation providing funding to NOAA for supporting weather prediction research and development.

Importantly, Congress, with full bi-partisan support, passed new legislation that set up an “Earth Prediction Innovation Center” focused on “advancing weather modeling skill, reclaiming and maintaining international leadership in the area of numerical weather prediction, and improving the transition of research into operations.”

Congress intended the EPIC center to be a robust, independent entity that would bring together the entire U.S. weather research community to build the best weather prediction system in the world.  To FINALLY, break through the divisions and duplication that are crippling U.S. efforts. And Congress gave NOAA millions of dollars to make things happen.

But as I shall describe, some in the NOAA bureaucracy has installed roadblocks in the way of EPIC.  

NOAA is about to make a big decision regarding EPIC:  will NOAA work with the UCAR/NCAR--the weather academic community-- to create it a viable center that will transform U.S. weather prediction? Or will the current stagnation continue?

Leaders in my community testifying in Congress about U.S. weather prediction

NOAA's Big Decision

NOAA is about to make a decision that will determine whether U.S. operational numerical prediction is second or third rate, or will become the best in the world.

As noted above, Congress has authorized and funded an EPIC center that would promote cooperation and innovation, and gave the responsibility to NOAA to make it happen.  

NOAA put out a request for proposals (RFP) for EPIC, but instead of following the will of Congress to create an independent center that would bring the entire community together to create the best global modeling system in the world, they altered the effort into a support services contract with NOAA.   Science-development or scientific research are not even in the document.  Bringing the community together is not in the document.  Innovation is not in the document.  NOAA retained complete control.  The intent was pretty transparent.

Now a major hope for many of us was that UCAR/NCAR, which represents the research community, would win the EPIC contract, reducing the chasm between NOAA and the academic community (UCAR/NCAR) and bringing the two groups together.  NOAA personnel made EPIC as unattractive as possible to UCAR/NCAR, but miracle upon miracle, UCAR/NCAR did put in a proposal for EPIC, hoping that a small start might grow into the national center that Congress had in mind.  And let me be frank, there are others in UCAR/NCAR that want EPIC to fail or go elsewhere, allowing them to continue to do business in the same way.

During the next few weeks NOAA will decide on who gets the EPIC contract--and this decision may well decide the future of U.S. weather prediction.  UCAR is the obvious choice, with deep experience in model development, relevant software engineering, and in supporting outside users.  No one is even close to their capabilities and experience.

This figure shows the skill of the U.S. global modeling versus other major groups

If NOAA selects UCAR/NCAR, the U.S. weather modeling community can follow a new, and far better path.  But if NOAA selects some beltway bandit enterprise or some consortium of less experienced academic institutions, then the future of U.S. operational prediction will not be bright.

In many ways, this is NOAA's last chance to get global modeling right.  Private sector firms, like IBM, are starting to building global modeling capacity.  The competition (like the European Center) are surging ahead.  NOAA. should be able to create a global modeling system that is the best in the world....not for some jingoist or nationalistic reason... but because better weather prediction would be of great benefit to humanity, providing a potent tool to protect mankind against a range of weather threats.