Sunday, January 26, 2020

The Future of U.S. Weather Prediction Will Be Decided During the Next Month

During the next few weeks, leadership in NOAA (the National Oceanic and Atmospheric Administration (NOAA) and the National Weather Service (NWS) will make a key decision regarding the future organization of U.S. numerical weather prediction.  A decision that will determine whether U.S.  weather forecasting will remain third rate or advance to world leadership.   It is that important.


Specifically, they will define the nature of new center for the development of U.S. numerical weather prediction systems in a formal solicitation of proposals  (using something called a RFP--Request for Proposals).

This blog will describe what I believe to be the essential flaws in the way NOAA has developed its weather prediction models.  How the U.S. came to be third-rate in this area, why this is a particularly critical time with unique opportunities, and how the wrong approach will lead to continued mediocrity.

 I will explain that only profound reorganization of how NOAA develops, tests, and shares its models will be effective.  It will be a relatively long blog and, at times, somewhat technical, but there is no way around that considering the topic.  I should note that this is a topic I have written on extensively over the past several decades (including many blogs and an article in the peer-reviewed literature), given dozens of presentations at professional meeting, testified about  in Congress, and served on a number of NOAA/NWS advisory committees and National Academy panels dealing with these issues.

The Obvious Problems

As described in several of my previous blogs, U.S. numerical weather prediction, the cornerstone of all U.S. weather prediction, is behind other nations and far behind the state-of-the-art.   Our global model, the GFS, is usually third or fourth ranked; behind the European Center and the UK Met Office, and often tied with the Canadians.

We know the main reason for this inferiority:  the U.S. global data assimilation system is not as good as those of leading centers.  (data assimilation is the step of using all available observations to produce a comprehensive, physically consistent, description of the atmosphere).

The U.S. seasonal model, the CFSv2, is less skillful than the European Model and is aging, while the U.S. is running a number of poorly performing legacy modeling systems (e.g., the NAM and Short-Range Ensemble System).  Furthermore,  our global ensemble system has too few members and lacks sufficient resolution.  The physics used in our modeling systems are generally not state-of-the-art, and the U.S. lacks a large, high-resolution ensemble system capable of simulating convection and other small-scale phenomena. Finally, operational statistical post-processing, the critical last step in weather prediction, is behind that of the private sector, like weather.com or accuweather.com. 

The latest global statistics for upper air forecast skill at 5 days shows the U.S. in third place.

There is one area where U.S. numerical weather prediction is doing well:  high-resolution rapid refresh weather prediction.  As we will see there is a reason for this positive outlier.

The generally inferior U.S. weather modeling is made much worse by NOAA's lack of computer resources.  NOAA probably has 1/00th of what they really need, crippling NOAA's modeling research as well as its ability to run state-of-the-science modeling systems.

Half-way Steps Are Not Enough

Although known to the professional weather community for decades, the inferiority of U.S. weather prediction become obvious to the media and the general U.S. population during Hurricane Sandy (2012), when the European Center model provided a skillful forecast days ahead of the U.S. GFS.  After a number of media stories and congressional inquiries, topped off by a segment on the NBC nightly news about abysmal state of U.S. weather prediction (see picture below), NOAA/NWS leadership began to take steps that were funded by special congressional budget supplements.


New computers were ordered (the U.S. operational weather prediction effort previously possessed only had 1/10th the computer resources of the Europeans), an improved hurricane model was developed, and NOAA/NWS began an effort to replace the aging U.S. global model, the GFS.   The latter effort, known as the Next Generation Global Prediction System --NGGPS, included funds to develop a new global model and to support applicable research in the outside community.

During the past 8 years, there has been a lot of activity in NOAA/NWS with the goal of improving U.S. weather prediction, and some of it has been beneficial:

  • NOAA management has accepted the need to have one unified modeling system for all scales, rather than the multitude of models they had been running.
  • NOAA management has accepted the idea that the U.S. operational system must be a community system, available to and used by the vast U.S. weather community.
  • NOAA management has increased funding for outside research, although they have not done this in an effective way
  • NOAA has replaced the aging GFS global modeling system with the more modern FV-3 model.
  • NOAA has made some improvements to its data assimilation systems, making better use of ensemble techniques.
  • Antagonistic relationships within NOAA, particularly between the Earth System Research Lab (ESRL) and the NWS Environmental Modeling Center (EMC) have greatly lessened.


But with all of these changes and improvements in approach, U.S. operational weather prediction run by NOAA/NWS has not advanced compared to other nations or against the state-of-the science.  We are still third or fourth in global prediction, with the vaunted European Center maintaining its lead.  Large number of inferior legacy systems are still being run (e.g., NAM and SREF), computer resources are still inadequate, and the NOAA/NWS modeling system is being run by very few outside of the agency.

This is not success.  This is stagnation.

But why?  Something is very wrong.

As I will explain, the key problems holding back NOAA weather modeling can can be addressed (and quickly), but only if NOAA and Congress are willing to follow a different path.  The problem is not money, it is not the quality of NOAA's scientists and technologists (they are motivated and competent).  It is about organization.  

Let me repeat this.  It is all about ineffective organization.

With visionary leadership now at NOAA and the potential for a new center for model development, these deficiencies could be fixed.  Rapidly.

The REAL Problems Must be Addressed

So with substantial resources available, the acute need for better numerical weather prediction in the U.S., and the acknowledged necessity for improvement, why is U.S. numerical weather prediction stagnating?  There are several reasons:

1.  No one individual or entity is responsible for success

Responsibility for U.S. numerical weather prediction is divided over too many individuals or groups, so in the end no one is responsible.  To illustrate:

  • The group responsible for running the models, the NWS Environmental Model Center (EMC), does not control most of the folks that develop new models (located OUTSIDE of the NWS in NOAA ( the ESRL and NOAA labs).  
  • Financial responsibility for modeling systems is divided among several groups including OSTI (Office of Science and Technology Integration) and OWAQ (NOAA Office of Weather and Air Quality), and a whole slew of administrators at various levels (head of the National Weather Service, head of NCEP, head of EMC, NOAA Administrator, and many more).

U.S. weather prediction is not the best?  No one is responsible and fingers are pointed in all directions.

2.  The research community is mainly using other models, and thus not contributing to the national operational models.

The U.S. weather research community is the largest and best in the world, but in general they are NOT using NOAA weather models.  Thus, research innovations are not effectively transferred to the operational system.

The National Center for Atmospheric Research in Boulder, Colorado

Most American weather researchers use the weather modeling systems developed at the National Center for Atmospheric Research (NCAR), such as the WRF and MPAS systems.  They are well documented, easy to use,  supported by NCAR staff and large user community, with tutorials and annual workshops.  Time after time, NOAA has rejected using NCAR models, decided to go with in-house creations, which has led to a separation of the operational and research communities.   It was a huge and historic mistake that has left several at NCAR reticent about working with NOAA again.

There is one exception to this depressing story:  the NOAA ESRL group took on WRF as the core of its Rapid Refresh modeling systems (RAP and HRRR).  These modeling systems, not surprisingly, have been unusual examples of great success and state-of-science work in NOAA.

3.  Computer resources are totally inadequate to produce a world-leading numerical weather prediction modeling system.

NOAA currently has roughly 1/10 to 1/100th of the amount of computer resources necessary for success.  Proven technologies (like 4DVAR and high-resolution ensembles) are avoided,  ensembles (running the models many times to secure uncertainty information) are low resolution and small, and insufficient computer resources are available for research and testing.

Even worse, NOAA computer resources are very difficult for visitors to use because of security and bureaucracy issues, taking the better part of a year, if they are ever allowed on.

There is a lot of talk about using cloud computing, but there is still the issue of paying for it, and cloud computing has issues (e.g., great expense) for operational computing that requires constant, uninterruptible large resources.


With responsibility for U.S. numerical weather prediction diffused over many individuals and groups, no one has put together a coherent strategic plan for U.S. weather computing or made the case for additional resources.    Recently, I asked key NWS personnel to share a document describing the availability and use of NWS computer resources for weather prediction:  no such document appears to exist.

4.  There is a lack of careful, organized strategic planning.

NOAA/NWS lacks a detailed, actionable strategic plan on how it will advance U.S.  numerical weather prediction. How will modeling systems advance over the next decade, including detailed plans for coordinated research and computer acquisition.  Major groups, such as the European Center and UKMET office, have such plans.  We don't.   Such plans are hard to make when no one is really responsible for success.

NOAA has tried to deal with the lack of planning by asking  U.S. researchers to join committees pulling together a Strategic Implementation Plan (SIP), but these groups have been of uneven quality, have tended to produce long laundry lists, and their recommendations do not have a clear road to implementation.


5.  The most innovative U.S. model development talent is avoiding NOAA/NWS and going to the private sector and other opportunities.

U.S. operational weather prediction cannot be the best, when the best talent coming out of our universities doesn't want to be employed there.  Unfortunately, that is the case now.  Many of the best U.S. graduate students do not want to work for NOAA/NWS--they want to do cutting edge work in a location that is intellectually exciting.

EPIC:  The Environmental Prediction Innovation Center

Congress and others have slowly but surely realized that U.S. numerical weather prediction is still in trouble want to deal with this problem.  To address the issue, Congress passed recent legislation (The National Integrated Drought Information System Reauthorization Act of 2018 ), which instructs NOAA to establish the Earth Prediction Innovation Center (EPIC) to accelerate community-developed scientific and technological enhancements into operational applications for numerical weather prediction (NWP).  Later appropriation legislation provided funding.

Last summer, NOAA held a community workshop regarding EPIC and asked for input on the new center.  There was strong support, most participants supporting a new center outside of NOAA.  The general consensus:  it will take real change in approach to result in real change in outcome.  They are right.
Two Visions for EPIC

There are two visions of EPIC and the essential question is which NOAA will propose in its request for proposals to be released during the next month.

A Center Outside of NOAA with Substantial Autonomy and Independence

In this vision, EPIC will be an independent center outside of NOAA.  It will be responsible for producing the best unified modeling system in the world, supplying the one point of responsibility that has been missing for decades.

This EPIC  center would maintain advisory committees that would directly couple to model developers, and should have sufficient computer resources for development and testing.   It would build and support a community modeling system, including comprehensive documentation, online support, tutorials, and workshops.

 Such a center should be in a location attractive to visitors and should entrain groups at NCAR and UCAR (like the Developmental Testbed Center).  It will maintain a vibrant lecture series and employ some of the leading model and physics scientists in the nation.


EPIC should be led by a scientific leader of the field, with a strong core staff in data assimilation and physics.  This EPIC center will be able to secure resources from entities outside of NOAA (although NOAA funding will provide the core support).

Such an EPIC center might well end up in Boulder, Colorado, the intellectual center of U.S. weather research (with NCAR, NOAA ESRL lab, University of Colorado, Joint Center for Satellite Data Assimilation, and more), and there is hope that UCAR (the University Corporation for Atmospheric Research) might bid on the new center.  If it did so and won the contract, substantial progress could be made in reducing the yawning divide between the U.S research and operational numerical weather prediction communities.


The Alternative: A Virtual Center Without Independence Or Responsibility

There are some in NOAA that would prefer that the EPIC center would simply be a contractor to NOAA that supplies certain services.  It would not have responsibility for providing the best modeling systems in the world, but would accomplish NOAA-specified tasks like external support for the unified modeling system and fostering the use of cloud computing.   It is doubtful that UCAR would bid on such a center, but might be attractive to some "beltway bandit" entity.   This would be a status-quo solution.

The Bottom Line

From all my experience in dealing with this issue, I am convinced that an independent EPIC, responsible for producing the best weather prediction system in the world, might well succeed. It is the breakthrough that we have been waiting for.

Why?  Because it can simultaneously solve the key issues that have been crippling U.S. operational numerical weather prediction centered in NOAA:  a lack of single point responsibility, that complex array of too many players and decision makers,  and the separation of the research and operational communities, to name only a few.

A NOAA-dependent virtual center, which does not address the key issues of responsibility and organization, will almost surely fail.

And let me stress.  The problems noted above are  the result of poor organization and management.  NOAA and NWS employees are not the problem.  If anything, they have been the victims of a deficient organization, working hard to keep a sinking ship afloat.


The Stars are Aligned

This is the best opportunity to fix U.S. NWP I have seen in decades.  We have an extraordinary NOAA administrator (Neil Jacobs) for whom fixing this problem is his top priority (and he is an expert in numerical weather prediction as well).  The nation (including Congress) knows about the problem and wants it fixed.  The President's Science Advisory (Kelvin Drogemaier) is also a weather modeler and wants to help.  There is bipartisan support in Congress.

During the next month, the RFP (request for proposals) for EPIC will be released by NOAA.  We will then know NOAA's vision for EPIC, and thus we will know whether this country will reorganize its approach and potentially achieve a breakthrough success, or fall back upon the structure that failed us in the past.


21 comments:

  1. Assuming that those in positions of authority make the decision to move forward with EPIC, what sort of timeline would we be looking at to see results?

    ReplyDelete
  2. Is this a decision that can be made without involving federal politicians? They are all pretty much busy right now. Trials. Plagues. Re-elections. Unfortunate passings of major sports celebrities. All of which are significantly more important than the actual day to day of governance.

    ReplyDelete
  3. Cliff
    Thanks for tracking this issue for us. I don't know of anyone else doing so in a public forum. Please keep us posted with how this situation evolves.

    For example:
    - What approach the NOAA RPF takes once it is published,
    - Timeline and characteristics for expected outcomes stated in the RFP
    - Your take on how the RFP aligns with future prospects.

    I thought that NOAA computer resources had just tripled in the last few years. Are you saying that even after recent upgrades they are still woefully deficient?

    In presentations to farmer groups last few weeks I've been pointing out some of the good news with respect to numerical weather monitoring in recent years:

    1. New GOES East and West platforms becoming operational in 2018 with 5x scan speed, 4x resolution, 3x wavelength bands than previous generation.

    2. COSMIC 2 satellite fleet launches in mid-2019 which when fully operational will lead to more accurate satellite monitoring of temperature, RH, air pressure and 10x the number of daily observations in the tropical belt.

    3. Upgrade to polarized dopper rain radar system from 2010-2013 providing more detailed information on type and rate of precipitation.

    4. And as you noted, transition in 2019 to FV3-GFS, with higher horizontal and vertical resolution planned for 2020.

    5. Continued progress in gridded weather databases such as URMA and NDFD, and incremental improvement on forecast accuracy. Tools like DAYMET for gridded climatic values and GDAS to supplement URMA.

    6. On the negative side, I have to agree that organization within the NOAA websites and GFS documentation is poor. Wither stuff does not exist or it is so difficult to find that you never find it, in which case it effectively does not exist.

    Note I am not a professional meteorologist, just a farm advisor who depends on hourly weather data observations and forecasts to track crop and pest development, fieldwork and spray conditions etc.

    So I am interested to hear if you think my enthusiasm about the developments above is misplaced.

    And thanks for not beating on the hardworking, underfunded folks at NOAA. Even from my outsider perspective, it is apparent that there are problems with funding and direction, due in part to supervision by politicians who do not appreciate the importance of the weather enterprise across many areas (including national security), and that NOAA staff competence and dedication are not the problem.

    ReplyDelete
    Replies
    1. Cliff (and you) get it right. The intentions and hard work are all there, but the results are typical of a bloated federal organization. It is always about leadership.

      Delete
  4. This is fascinating! Q about the RFP: Who's written the scope, and can the public see it?

    You have written often about the need for more (and better) observation data ...blind spots, as it were - like the coast in Oregon. Rigorous science thrives on feedback, testing ...proof. Is more observation (a robust mix) a part of this?

    Very important matter... exciting.

    ReplyDelete
  5. Cliff, Good blog and please keep us posted! I really want NOAA to make a bold move to take the U.S. to number one.

    ReplyDelete
  6. The HRRR is a generally a success, but some European and Asian countries run operational models at comparable or higher resolution, often as an ensemble. Granted, they have a smaller geographic region to cover, but nevertheless, the US lags in this area as well.

    ReplyDelete
  7. Hi Cliff. One reason, in my view, is that they excluded RAMS and associated expertise, in advancing weather prediction. Brazil has used BRAMS successfully yet leadership at NOAA has ignored this capability. This includes when WRF was developed. As you are perhaps aware, in cases where NAM and RAMS (later run in house at WSFOs) were compared, RAMS most of the time was more skillful. I have not seen such comparisons done recently as I assume NWS were discouraged or prevented from using RAMS.

    On the larger scale, Roni Avissar and Bob Walko have developed OLAM yet to my knowledge their expertise and knowledge have not been utilized by the NOAA.

    Together, the failure to utilize expertise and model development of RAMS, BRAMS and OLAM is an example of the lack of openness by NOAA leadership.

    Best Regards

    Roger

    ReplyDelete
  8. Sidebar to all of the complex and thoughtful points raised by Cliff: When it comes to severe local convection, the NWS, both at the SPC and in most of its local WFOs remains the world leader in short and medium range forecasting. Much of this is due to the excellent training programs in NWS and, as in the military, more "combat experience" in pattern recognition. Yes, this is only tangentially related to the blog content but, then again, it's evidence that NWS meteorologists often rise above the modeling deficiencies in the mesoscale. We can be thankful for that.

    ReplyDelete
  9. Cliff runs into the deep state. It is not really shocking to anyone that a government organization decides to ignore superior work in another organization (whether private or government) in order to grow their own budget. They want it in-house.

    NOAA, like all government organizations, exists primarily as a jobs program. It exists for itself.

    IF all we cared about was good predictions, we would give accuweather or the weather channel a $200 million dollar bonus for whichever one got it right the most times each year.

    But, that isn't the reason for NOAA.

    ReplyDelete
  10. Possibly because NOAA has focused on selling climate hysteria instead of their mission on providing accurate weather forecasts.

    https://wattsupwiththat.com/2020/01/15/while-noaa-nasa-claims-2019-as-the-second-warmest-year-ever-other-data-shows-2019-cooler-than-2005-for-usa/

    https://wattsupwiththat.com/2019/03/13/ca-sea-level-rise-alarmist-study-ignores-30-years-of-noaa-data-with-no-coastal-sea-level-rise-acceleration/

    https://wattsupwiththat.com/2018/06/25/nyt-noaa-may-be-stripped-of-climate-mission/

    https://wattsupwiththat.com/2018/02/20/noaa-caught-cooking-the-books-again-this-time-by-erasing-a-record-cold-snap/

    ReplyDelete
  11. Why would anyone expect scientific policy-making, including allocation of resources for scientific projects, to make any sense in the current administration? At least we can be thankful we have a new U.S. Space Force military organization. Perhaps we can find a way to weaponize our weather satellites.

    ReplyDelete
  12. Ignoring 1500 papers on solar forcing of the total atmospheric vertical column isn't helping either.

    ReplyDelete
  13. I like the EPIC recommendation, but would like to point out that even the “vaunted” ECMWF model has some serious issues. I noticed last summer (2019) at least a couple of times the ECMWF showed development of an intense tropical storm to near hurricane strength OVER LAND in the western bulge of Africa. The GFS-FV3 correctly showed only a weak low pressure system in both cases. I also remember seeing the ECMWF develop a weak tropical storm OVER LAND in South Texas/Northern Mexico once last summer as well. Needless to say, no other models showed intensification over land.

    ReplyDelete
  14. One neglected area where NOAA can blow the pants off other countries is in the area of statistical post-processing. We need to develop a Super MOS system.

    Unfortunately, the powers that be still don't realize a good MOS system will handily beat any kind of raw or ensemble NWP output. The powers that be also don't realize that continually tweaking an NWP model ruins the MOS. So here's the strategy:

    1) Take the latest & greatest three NWP models, freeze them, and run them retroactively on the last 15 years of data.
    2) Have MDL create MOS equations for each NWP model.
    3) The official Super MOS forecast is a consensus of the 3 individual MOS forecasts.
    4) When a new NWP model comes along, freeze it, run it retroactively on the last 15 years of data, develop a MOS system for that NWP model, and add it to the consensus mix.
    5) Only when an old NWP model fails to contribute predictive skill to the consensus mix should that old NWP model be retired.
    6) If NCEP decides to tweak an existing NWP model then continue to run the frozen version concurrently in the background so that the stable MOS forecasts can still be produced.

    The numerical forecasts produced from this system is guaranteed to beat anything else out there. Running new NWP models retroactively on a large 15 year historical dataset allows reliable single-station equations to be developed instead of those for large regions. Research has shown that NOAA screws up by retiring old NWP models too early even though their well developed MOS systems add predictive information to a consensus mix.

    No other country has a Super MOS system like this. We could be the first. I wanted to do this 20 years ago in a private venture with my colleagues from Penn State but ended up pursuing a different venture.

    ReplyDelete
  15. " to make any sense in the current administration?"

    Did the current administration make the decisions that have led to the issues that are explained in this post?

    I note the following:
    "During the next month, the RFP (request for proposals) for EPIC will be released by NOAA."

    This sounds as though the "current administration" is aware of and looking to improve.
    I've no personal knowledge, just reading what is in the post.

    ReplyDelete
  16. The current administration, if you want to call it that since its for the most part one person, will be gone in 2024 (or sooner but doubtful). However, science might do better with the profit motive and competition that only comes from divorcing itself from government handouts. The only thing the government really excels at is dysfunction and waste.

    ReplyDelete
  17. Forget Big Blue, go with Cray!

    ReplyDelete
  18. "The problems noted above are the result of poor organization and management. NOAA and NWS employees are not the problem. If anything, they have been the victims of a deficient organization, working hard to keep a sinking ship afloat"

    WOW, just wow! As a contractor with 20+ years on a NOAA site, I can say I feel "truer words have never been spoken". If only the managers would listen to what the workers have to say...

    Thanks for a great post - I keep my fingers crossed that the ship will stay afloat!

    ReplyDelete
  19. Suppose that outside-of-NOAA center does indeed produce the best modeling system in the world, and indeed is given all the resources needed for development and testing.

    It has still made absolutely no change to the model guidance going to forecasters. Only an operational implementation -- meaning inside NOAA -- can do that. And it has to run on a machine far smaller than was used in development, in operational, near real time, 24x7 mode, not research mode that can tolerate glitches.

    Further, by placing all the excellence outside of NOAA, there's no path to get back inside. Not even to OAR/ESRL, and certainly not to NWS/EMC.

    So there's an attractive ivory tower built in some attractive location, that proceeds in splendid academic isolation, developing a wonderful model -- that cannot be used in NOAA operations.

    ReplyDelete