Climate Versus Weather Prediction: Do We Need to Rebalance?

Last week there was a hearing before the House Subcommittee on the Environment regarding a piece of legislation, The Weather Forecasting Improvement Act of 2013, that has the goal to:

"To prioritize and redirect NOAA resources to a focused program of investment on near-term, affordable, and attainable advances in observational, computing, and modeling capabilities to deliver substantial improvement in weather forecasting and prediction of high impact weather events, such as tornadoes and hurricanes, and for other purposes."

Sponsored by the Republican side of the committee, the goal of this act is to provide sufficient resources and guidance to push weather prediction forward more rapidly.  And the sponsors are right, with better organization and enhanced resources, weather prediction could improve rapidly in the U.S..  The Democratic side of the committee were clearly unhappy since it is clear that part of the intent of the bill is to rebalance resources in NOAA:  more for weather prediction and less for climate.

Is their an imbalance between support and priority for weather prediction and climate in the U.S.?

My take: although this bill has its issues and needs serious revision,  I believe that resource allocation has become highly skewed towards climate prediction, to the detriment of BOTH weather prediction and understanding/prediction of climate change.  I also believe that a revised bill could be highly bipartisan and a major positive for the U.S. and the world for both weather prediction and climate.

Weather Prediction Versus Climate Prediction Versus Seasonal Prediction

For the sake of this blog, I will consider that weather prediction covers the period from 0 to two weeks.  Climate prediction more than a few years out.  And there is another animal that is rapidly developing (although skill is modest):  seasonal prediction from two weeks to a year out.

Weather prediction is heavily based on numerical weather prediction, in which one starts with a 3D description of the atmosphere and computer models forecast the detailed evolution of the atmosphere.

Climate prediction depends on climate prediction models (also known at General Circulation Models), which are VERY similar to weather prediction models except that the climate models let gas concentration change, use coarser resolution, and are connected to evolving ocean models.   Climate simulations are not used to provide a detailed description of the climate at one time, but to provide climate statistics (e.g., winters will be warmer, snowpack will be less, extremes will change in a certain way, etc.).  Climate simulation makes use of the fact that changes in "external" forcing--like varying greenhouse gases--will alter the weather patterns over the world.  Climate prediction also makes use of long-term cycles (such as the Pacific Decadal Oscillation (PDO) or the Atlantic Multidecadal Oscillation (AMO)).

And then there is seasonal prediction, which really is a hybrid between the two that uses both extended runs of weather prediction models (like the U.S. GFS) and an understanding of modes of natural variability (like El Nino/La Nina).   Seasonal prediction also makes use of statistical relationships (such as the relation between snowpack and soil moisture and the weather over the next few weeks and months).

The Imbalance

As described in my previous blogs, there is a substantial imbalance in the resources available for weather and climate prediction and research.  For example, the computer resources available for climate is at least one-hundred times larger than for weather prediction (see graphic below):

And there is also a HUGE difference in the amount the U.S. government spends on climate and weather research.  For example, President Obama's request for climate research is roughly 2.7 billion for the U.S. Climate Change Program, while according to the Federal Coordinator for Meteorology that total budget for weather research (mainly in NASA!) was 1.25 billion.   So roughly 2.5 times the money for climate research (and the ratio is really much more because most of the NASA funds are used for supporting satellite missions and climate work).

As a researcher at a leading university that does both climate and weather research, let me assure you it is MUCH easier to get funding for climate research than weather prediction-related research.  NSF, NASA, NOAA and many agencies have big money for climate research, while NOAA is constantly zeroing out their funding for extramural weather research (like their CSTAR program).  A real eye opener.

The imbalance is profound and large.  Does this make sense?

Why Advances in Climate Prediction and Warning Depend on Weather Forecasting

Some supporters of this imbalance argue that climate and weather research are inter-related and thus one should not be concerned about the budgetary issues.  That climate and weather research is on a continuum that cannot be divided in any meaningful way.  But I would argue that how money is prioritized does make a huge difference and that in actuality the ability to predict the future climate DEPENDS on weather prediction.  Let me make the case.

Weather prediction and climate prediction both depend on the same technology:  numerical models of the atmosphere.   In fact, both the resolution and physics used in climate and weather prediction models are converging rapidly.   Weather prediction models are run several times a day and are rigorously verified against a range of OBSERVATIONS.   Thus, working on weather prediction allows a cycle of continuous verification and improvement.  When you run a climate model into the next century, verification is an obvious problem. And a fact that is often buried is that climate models are often tuned to match the contemporary climate and thus their predictions are suspect.

Bottom line: if you want better climate predictions, you need better numerical models of the atmosphere and weather prediction offers the fastest and most effective route to better models.

The output from climate and weather prediction models are indistinguishable

Weather Prediction as the First Line of Defense For Climate Change

There are many that believe that the weather may become more extreme under global warming (although I think there are a LOT of uncertainties in this assumption).  But let us assume that extreme weather events (hurricanes, floods, tornadoes, heat waves, etc.) WILL become more extreme and frequent in the future.  What is the first thing you need with more extreme weather?  BETTER WEATHER FORECASTS.  Good weather forecasts have a huge impact in saving lives and property for extreme weather events, something shown for Superstorm Sandy where about 150 died in a region of tens of millions of people (and many who died did so because they ignored the warnings).   So if you care about the impact of climate change, you should be an enthusiastic supporter of better weather prediction.

And if you are interested in the local implications of climate change, the  best technology is dynamical downscaling, in which weather prediction models (such as the Weather Research and Forecasting Model, WRF) are driven by global climate models.  Thus, cutting edge climate prediction depends on weather prediction models.

The Impacts of Weather Prediction are Large and Certain, while Climate Prediction Impacts are Uncertain

I don't think I have to work hard to convince you that improved weather prediction saves lives and improves the economy TODAY.  With trillions of dollars of U.S. economic activity sensitive to the weather, even small improvements can save our nation tens of  billions or more a year.  The U.S. has some of the most extreme weather in the world and we require state-of-the-art weather prediction to protect our citizens and to aid economic activity.  High-resolution U.S. weather prediction is only done by the U.S. government or U.S. companies/universities--the Europeans will not do this for us.  But dozens of major groups around the world are doing state-of-the-art global climate predictions for next century.  Quite frankly, if the U.S. wasn't doing climate simulations there would still be plenty to choose from and lots of research activities using them. And the skill of such climate predictions are uncertain and few nations appear to be willing to make major economic decisions based on them.

Let me be clear:  I believe global warming is a threat to mankind and that research is needed on this topic.  But weather forecasting research is certainly equally as important, and in fact serves climate prediction and warning as well.

Bottom Line

The framers of the Weather Forecasting Improvement Act are correct:  the U.S. must put more resources into weather prediction research, development, and infrastructure.   We have underinvested in weather prediction, with very negative effects on the nation.

The payback on more investment in weather prediction for the nation would be huge.  And, it would greatly enhance our ability to predict and warn our nation for any impacts of climate change.  If the money existed, it would be nice to greatly increase weather prediction support and leave climate alone.  But if resources are tight, the only logical decision is to rebalance our current investments more towards weather prediction.

The current bill needs some work (more on that in a future blog) and I truly hope it can become bipartisan--and it should.   Better weather prediction is a win-win situation, both for climate and short-range weather forecasting.

A Revised Weather Improvement Act That Would Radically Improve U.S. Weather Prediction

As I noted in an earlier blog, the House Science Committee is now considering the Weather Forecasting Improvement Act of 2013.  There are some good things in this bill, some problems that need to be addressed, and several critical issues that are simply not considered.

I believe strongly that an an improved and revised bill could potentially greatly improve weather prediction in the U.S. and could be supported by both sides of the aisle.  In fact, I can think of few subjects that could more readily garner bipartisan support in Congress.

So let me go out on a limb and suggest the outline of a bill that could revolutionize U.S. weather prediction. 

The Bill's Central Goal

The current bill has a goal

"To prioritize and redirect NOAA resources to a focused program of investment on near-term, affordable, and attainable advances in observational, computing, and modeling capabilities to deliver substantial improvement in weather forecasting and prediction of high impact weather events, such as tornadoes and hurricanes, and for other purposes."

I agree with the bill's authors that that the balance between the resources available for weather forecasting and climate is out of whack (too little for weather forecasting), but the term "prioritize and redirect" has gotten the Democrats and climate folks nervous.  And the list of high impact events and phenomena is too limited.     NOAA is only part of the problem/solution, and the bill leaves out the crucial area of communication, dissemination, and application of forecasts.  A little wordsmithing and modification can fix these issues:

"To establish a focused program of investment on affordable and attainable advances in observations, computing, modeling, communication, and dissemination capabilities to deliver a substantial improvement in the quality and application of weather forecasting and prediction information in the United States, with an emphasis on high impact weather events, such as severe thunderstorms, floods, heat waves, hurricanes, snowstorms and windstorms.  NOAA and other Federal agencies are directed to insure that U.S. weather prediction and dissemination capabilities are state-of-the-science.

More balanced and something everyone can agree to.

 Solve the Number One Problem-- the Lack of Community Participation, Interaction and Feedback

The central reason that U.S. prediction capabilities lag others around the world and falls far short of our inherent capabilities is not the lack of big computers (although that hurts!).  The real problem has been the isolation of the National Weather Service from the nation's research and development community, as well as inadequate interactions with the private sector and users communities.  The U.S. has the largest weather research community and the most extensive private sector, yet the NWS has been really poor in tapping this expertise and taking advantage of the knowledge of those outside of NOAA.   A number of U.S. National Academy advisory committees has recommended the establishment of a  NWS Advisory Committee, but NWS management has stubbornly and counterproductively refused.   A not-invented here, we do not need to get advice from others, attitude has pervaded the National Weather Service and NOAA.

This bill could deal with these issues in two ways:

(1)  The establishment of a National Weather Service advisory committee comprised of experts in modeling, forecasting, communication, observations, and dissemination. This committee would encompass experts from the research community, private sector community, and others.  This would only require minimal funding.

(2)  The establishment of a higher level advisory group the encompasses the users and producers of weather information, as well as the research community.  One possible model for this is the proposed Weather Commission (proposed by the U.S. Weather Coalition, here).  This would be a congressionally mandated group, with basic support for meetings and reports.

A National Weather Research Program

The current bill sets up a fund for forecasting research and innovation, which is important and needed.  The NWS has not invested sufficiently in weather prediction research. But some of the bill details are problematic.  First, the identification of issues is too narrow, with great emphasis on tornadoes and hurricanes.  These are very important issues, but so is flooding, winter storms, wildfires, midlatitude cyclones, heat waves, droughts, transportation weather, and others.  The language needs to be broadened to encompass all important weather phenomena.

An important missing element of this bill is the lack of support for research and development regarding weather information communication and dissemination.  The recent House hearing underlined that some members of the Environment subcommittee did not appreciate the importance of social scientists and others examining the communication issue.  Let me be clear now:  creating good forecasts is only half the job...the easier one.  Creating products that clearly provide information and promote the correct response is just as important, and the NWS has not invested sufficient resources in this area.    Several people died during the El Reno tornado unnecessarily, because they decided to hit the roads when then tornado warning went out.

Second, the funding is provided to NOAA's Office of Atmospheric Research (OAR) for distribution.  This is a major deficiency.  A major part of the U.S. weather prediction problem is that the group doing operational numerical weather prediction (the National Weather Service, mainly the Environmental Modeling Center) does not control the group that is responsible for weather research (in NOAA OAR).    NOAA OAR does some good work (mainly on the Rapid Refresh forecasting system and data assimilation), but it also does all kinds of work that never gets into operations. Relations between NOAA OAR and NWS operational folks is often tense and inefficient. (A really good example of this is that last summer a very highly placed individual in OAR told me that I was wasting my time trying to get more powerful computers for the NWS...I should get them for his group, which would do quasi-operational weather prediction). Another issue is that funds given to NOAA OAR will tend to stay within NOAA and never reach the critical research community outside of NOAA (trust me, I know about this from experience!).

The bill can help with this.   The funds should  go to a new National Weather Research Program office in either the National Weather Service or NOAA, and the bill should be EXPLICIT that the bulk of the funds (say 60-70%) be directed to extramural funding outside of NOAA.  I can not  exaggerate the importance of this.  Furthermore, at least 20% of the funds should go for forecast dissemination and communication research and development.

The bill should be explicit that the NWS advisory committee be given a large role in deciding the priorities of this new funding, in concert with those responsible for operational weather prediction in the NWS.

Finally,  the bill goes overboard in micromanaging this research.  Don't call out a tornado research program or narrowly define the weather features of concern.  

Observing System Planning

This bill takes on a critical issue, finding a rational way to design the U.S. and global weather observing system.  It is outrageous that NOAA has spent billions of dollars on weather satellites and observing systems without using rational tools for deciding how to design the best observing system for the least amount of funding.   Duplicative and wasteful observing system acquisitions have been all too frequent, billions have been wasted on poor decisions regarding satellite systems, and extremely valuable, cost effective data sources (e.g., aircraft data from commuter planes, COSMIC-2 GPS satellite data) are not acquired.  A total mess and a severe indictment of NOAA management.

This section of the bill starts out well, asking NOAA to

1) develop and maintain a prioritized list of observation data requirements necessary to ensure weather forecasting capabilities to protect life and property to the maximum extent practicable;
(2) undertake ongoing systematic evaluations of the combination of observing systems, data, and information needed to meet the requirements developed under paragraph (1), assessing various options to maximize observational capabilities and their cost-effectiveness;
(3) identify current and potential future data gaps in observing capabilities related to the requirements under paragraph (1); and
(4) determine a range of options to address gaps identified under paragraph (3).

But then it goes into super micromanagement mode, calling for only one (of many) approaches to doing such an evaluation (OSSEs--Observing System Simulation Experiments).   Congress MUST hold NOAA's feet to the fire on this, considering the long-history on inaction, but the selection of tools must be left to the experts.  The key for this bill is to set the priorities, assign responsibilities and deadlines, and to provide the resources to get the job done. 

Computing Resources Prioritization Report

This section of the reports requests NOAA to provide a report describing how the U.S. will acquire and maintain the computer resources required for state-of-the-art weather prediction.  NOAA management has consistently allowed the U.S. to fall behind in numerical weather prediction computing and only pressure from Congress seems able to gain their attention to this issue.  But Congress must demonstrate its readiness to invest in necessary weather prediction computation and to allow the NWS to access the huge computer resources available for climate research and simulation.

Commercial Weather Data

Some members of this committee became enthusiastic about using more private sector weather data, particularly for satellite information.  This is partially driven by the relative incompetence of NOAA satellite acquisition and the intense lobbying of some private sector companies wanting to get into the GPS satellite business.  This section of the bill is too satellite oriented and assumes that private sector firms are always the best deal.  For example, there is extraordinarily valuable data from commercial commuter aircraft (TAMDAR) that the NWS is not acquiring, data that has been proven to substantially improve weather forecasts.  Second, for some satellite systems (GPS satellites) it might be that international cooperative efforts (e.g., COSMIC-2) might be far more cost effective and more quickly available.  The bill should make it clear that the NWS should report back on the potential for commercial data sources of all types and not make any assumptions of the best course until the information is in.

What is Left?

The above changes would lead to significant improvements in weather prediction in the U.S. and set the nation on a more rational, cost-effective, course for enjoying such improvements.   A huge step forward.

Although the new act would promote weather forecasting research, the real challenge is to encourage the most promising research and development, and then effectively moving that into operations.  Some approaches that might help in this:

(1) Combine the weather forecasting research groups in NOAA OAR and the operational weather prediction group in the NWS into one entity controlled by one manager.  The current structure is clearly dysfunctional.  The new group might be moved into the National Weather Service, perhaps being located in Boulder, Colorado--the intellectual capital of U.S. weather prediction.

(2)  New organizational structures that promote integrated research and operations, and a new attitude towards working with the research community and being open to innovation, are needed.  I have ideas on how this could be done, but that will wait until another blog.

(2)  To improve weather prediction, it is important to have a neutral arbiter of quality, a group that could evaluate potential innovations in a operational-like environment.   The Developmental Testbed Center (DTC) in Boulder, Colorado should/could play this role if given the chance and funding.

This blog is already too long...but the bottom line is clear:  improved weather prediction could be a huge positive for the U.S. and this bill, properly revised, could go a long way to making it happen.