Tuesday, July 31, 2018

Addressing the Stagnation of U.S. Operational Numerical Weather Prediction

U.S.  operational numerical weather prediction has stagnated.
  • For decades, the skill of the U.S. global model has lagged those of leading international weather prediction centers
  • U.S. operational numerical weather prediction (NWP) has fallen behind the state-of-the-science in other areas as well, even though the U.S. is the leader in meteorological research.
  • Regional prediction, particularly ensemble-based probabilistic forecasting, is lagging far behind where it should be.
  • And as easily demonstrated, U.S. operational weather prediction is not catching up.

But why the stagnation?

It is not for lack of resources.  It is not for the lack of sufficient personnel.  It is not because NOAA/National Weather Service meteorologists or leadership lack interest, knowledge, or motivation.  It is not because the research community does not want to assist.

It is because the structure of research, development, and operations in the U.S. is essentially broken and dysfunctional.  Divided and inefficient, with poor organization.  Lack of strategic planning. And because the operational and research communities are not working together effectively.

The cost of inferior weather prediction to the nation is immense.  Property and lives lost unnecessarily due to poorly predicted storms.  Economic loss and inefficiency resulting from weather prediction errors that could be substantially reduced.

The bottom line of this blog:

U.S. weather prediction will never become state-of-the-art until the U.S. governmental weather prediction enterprise is totally reorganized.

And in this blog I will describe how it could be reorganized to address the inherent problems, allowing U.S. operational prediction to be the best in the world within a few years.


The opportunity

Today there is a great opportunity to fix the problem.  The first thing needed to address a problem is to know that it exists.  The public knows, because there have been many stories in the national media on the trailing U.S. weather prediction enterprise.  Congress knows, and even passed a bill (the U.S. weather research and innovation act of 2017).  NOAA/NWS management knows and have started some well-motivated, but ineffective measures.   Private sector forecasting firms know and have pushed for reforms.  One U.S. National Academy of Sciences report after another has called for change.

Today we have new and highly able high-level managers in NOAA  (administrators Neal Jacobs and Admiral Tim Galludet) that understand the problems and would like to fix them.  We have an administration that wants "to make America great again."

 Incremental changes that leave current structures in place will fail.   After decades of talking about the problem, can we finally take the substantial measures that will make a real difference?

The warning signs

I have published at least a dozen blogs documenting U.S. inferiority in operational numerical weather prediction, the foundation of all weather forecasting in the U.S.  Others have described the problems as well.  But let me show you a few examples that demonstrate the problems.

Consider the 5-day forecast over the Northern Hemisphere for a typical level aloft (here, 500 hPa, about 18,000 ft, half the atmosphere is above and below this level).  Consider a measure of forecast error (root-mean-square error, RMSE) for the U.S. model (the GFS, red line) and the top world model (the European Center, ECMWF, black).  The  top plot below shows the forecast errors from 1996 to today, while the bottom one displays the difference between U.S. and European Center errors.

Note that the U.S. model (red line) has a higher error that the ECWMF at all times.  And the U.S. is not catching up.  In fact, our standing compared to ECWMF has worsened in the past few years.
 Another sign of NOAA/NWS problems is inferior statistical post-processing of model forecasts, a step in which several model forecasts can be combined, biases removed, and the other improvements.  The U.S. private sector uses much more sophisticated approaches (which they secured though cooperation with the university research sector, including the National Center for Atmospheric Research, NCAR). 

Here is an example of forecast accuracy from Chicago from the forecastadvisor website (virtually every other city shows a similar situation).  Major private sector firms (e.g., the Weather Channel), using some of the same model inputs, have much better forecasts that the NWS.


One national academy report after another (e.g., this one) has recommended the U.S needs a convection-allowing, high-resolution ensemble system that is large enough and designed sufficiently well to provide useful forecast uncertainty information.   The NWS has lagged in this area and has only created a kluged a small ensemble (the Storm Scale Ensemble of Opportunity, SSEO, now HREF).  Not good enough.

During the past several years, the NWS Short-Range Ensemble Forecast System (SREF) has stagnated in resolution and design.  Same with the Global Ensemble (GEFS).  The seasonal forecasting system (CFSv2) has remained basically unchanged for years, with no plans for upgrading for many years.

The only exception to this dismal situation is the excellent work done by the NOAA ESRL rapid refresh team  for frequently updated, short-term forecasts (RAP, HRRR).

The NWS is developing a new global modeling system (FV-3), but there is no evidence that it will significantly improve forecasts without better data assimilation and physics (see figure below for comparison of the 5-day forecast).

I could list a dozen other examples of ways U.S. NWP has fallen behind, and virtually everyone who knows the situation would acknowledge that U.S. numerical weather prediction is woefully lagging where it should be.

Poor Coordination and Lack of Cooperation, Coupled with Insufficient Planning and Organization

So how does a nation with leading weather scientists, leadership in computer science and information technology, and substantial vulnerabilities to storms and other weather features (e.g., hurricanes, windstorms, severe thunderstorms, wildfires) end up with second rate forecasting capabilities?

Poor organization, lack of cooperation, acceptance of inferiority, among others.

You can think of U.S. numerical prediction as an old tree that has been left without pruning, trimming, or planning:  its eventually becomes very overgrown and unhealthy over time.

A visual metaphor of U.S. operational NWP

Consider the poor organizational structure of U.S operational numerical weather prediction.  Keep in mind that the National Oceanographic and Atmospheric Administration (NOAA) is part of the Dept. of Commerce, and that the National Weather Service is part of NOAA.

Operational weather modeling is the responsibility of the Environmental Modeling Center (EMC) of the NWS National Centers for Environmental Prediction (NCEP).  But they don't run the models--they let another group in NCEP do it (NCEP Central Operations, NCO).   Now, although EMC is responsible for the weather modeling, they don't develop most of their own models.   That is done by a number of other centers, some in the NWS and some not.  GFDL in Princeton develops some (hurricane models, new global model), NOAA ESRL (not in the NWS) develops others (e.g., rapid refresh models).  The statistical postprocessing is developed outside of NCEP in MDL (Meteorological Development Lab).  Decision making for new models (and the financial management) is done by different folks outside of NCEP  (Office of Science Technology Integration, OSTI).  No one individual or group is responsible for U.S. operational weather research, operations, and research.  It would be hard to make up a more ineffective approach to managing such a complex task.
Even more inefficient, consider that other agencies (like NASA, the Navy, and the Air Force) are all running their own (and different) numerical weather prediction systems, with independent development groups.

Not bad enough?   The research community through the National Center for Atmospheric Research (NCAR) has developed a separate suite of weather and environmental modeling systems, from the global to local scales. 

Worse?   A NOAA Office of Water Prediction and a National Water Center was set up in Alabama to take on the nation's water prediction (and hydrological forecasts), as if that should be separated from the rest of water prediction (this boondoggle was the result of a Senator wanting some pork for his district).

Vast sums are spent on uncoordinated weather forecasting research by the National Science Foundation, NOAA, DOD, NASA and others, much of redundant or never destined to assist operational capabilities.

And did I mention that there is no concrete strategic plan in NOAA/NWS for the development of U.S. environmental prediction?   Just vague platitudes.   This is in stark contrast to the detailed planning by NOAA/NWS's big competitors (e.g., the European Center, the UKMET office, etc), which as extensive, detailed, and coherent planning processes.

The bottom linewe have a disastrously inefficient system of developing, running and post-processing numerical weather prediction guidance for the U.S., with agencies doing their own things, disorganization growing, and U.S. capabilities stagnating.

Most decision makers are thinking about protecting their own turf and resources, and few high-level NOAA administrators have known enough to see the problem or were willing to take the heat for the necessary changes.  Numerical weather prediction is technically very challenging and few in Congress has sufficient background to revamp the organization of U.S. weather forecasting.


The latter point is worth repeating:  numerical weather prediction is perhaps the most complex activity of our species, involving billion dollar satellites, global data collection, running complex models that encompass molecular to global scales, and that use the largest computers on the planet.  You can't do this in a haphazard, uncoordinated away as done in the U.S. and expect to be state-of-the science.

The current NOAA/NWS response will not solve the problem

NOAA/NWS management know there is a big problem, and to be honest, some things have improved.  A few years ago, there was outright tension between model developers in NOAA ESRL (Earth System Research Lab in Boulder) and the NWS EMC folks.  That is much better now.  The NWS is putting more funds into extramural research in the academic community and is holding workshops and meetings to gain input.  Recently, the NWS begin developing a new global model (based on the GFDL FV-3 system) to replace the hoary old model (the GFS).

NOAA/NWS has not dealt with the central problem of poor organization, coordination and planning.

They have not created a structure to develop actionable strategic plans.  Different folks are responsible for model development and operations.  The  NOAA extramural funding has often not been well spent.  The University community is drifting away, with nearly all using the NCAR models.  There is  insufficient computer resources to support weather prediction and weather prediction research that the nation requires. And there is little evidence that the new global model will improve verification scores very much, since the real problems are in data assimilation, physics, ensembles, and post processing.

The National Center for Atmospheric Research Mesa Lab
The NWS has held workshops and organized all kinds gatherings to garner support, but these have led to long laundry lists, with little ability to lead to actionable, organized efforts.

And let me be clear that I am not singling out NOAA management for this problem.  It is difficult for a line manager in NOAA to visualize and change a complex organizational structure.   Important partners (like NCAR, DOD, NASA) have often shown little interest in working with NOAA, wanting to maintain their own modeling sandbox. Congress has not been sufficiently attentive to the dysfunctional structures they have created and maintained.

How to fix the problem of U.S. numerical weather and environmental prediction

The key is reorganization, coordination and planning.  The best possible circumstance would be to bring all environmental prediction activities of NOAA into one group-- research, development, operations--with one individual responsible for the whole operation.  We are talking about combining the NWP activities and responsibilities of NWS NCEP, NWS MDL, NOAA ESRL, NOAA GFLD, the National Water Center, and NWS NCO (and others) into one group.  Even better would be to create a truly national center, and bring in the NASA and DOD components--but that is a heavy lift.

But let's imagine something that is smaller lift, that might be a good first step.  Leave responsibility for running the weather/environmental prediction models in the National Weather Service, but move all model development and testing to a new integrated entity within NOAA:  The National Environmental Prediction Research and Development Center (NEPRDC).  A schematic of the organizational structure is shown below.  It would include a chief scientist to help organize and lead the scientific work, with a strong scientific advisory committee.

One organization in NOAA would be responsible for research, development, and testing of the nation's environmental models (weather, ocean, hydrological, coastal, etc.)  The center would have ample supercomputer resources for model testing (which does not exist now) and control of extramural funding, which would help support university (and other) researchers working in research that will directly address current and future modeling problems.

An ideal location for the center would be in Boulder, Colorado-- the center of U.S. weather research, with both NOAA ESRL and NCAR (the university community's center) located there today.   The Developmental Testbed Center (which can provide extramural support for U.S models and help evaluate the new ones, is already in Boulder). Some  NWS EMC personnel (who are now in DC) could work as a satellite center in Boulder.  Bringing NCAR  (also in Boulder) and NOAA together through such a center is critical, and Boulder is a far more central venue than DC for combining the nation's modeling efforts.

Improvements in environmental prediction will only come from sustained, coordinated hard work in physics, data assimilation, and other key areas.  This requires coherent planning and the coordination of the vast scientific and technical resources of the U.S.  The current system is incapable of such effort, the proposed one will be up to the task.

As a first step in fleshing out this proposal might be a national workshop on U.S. environmental prediction, bringing together the entire environmental prediction community, with a detailed white paper coming out of it.  And with so many changes in the bureaucratic structures, Congress will have to be involved.

But in the end, it is clear that the current structures, the result of legacy and administrative drift over decades, are failing.   Only major restructuring and reimagining of U.S. environmental prediction can result in the necessary changes.    The U.S. can easily regain leadership in weather prediction if we only have the will to acknowledge the current failed structures and replace them with something better. 

The U.S. has fallen behind in so many areas due to complacency, poor leadership, self-interest, and loss of energy.  Can we do better in weather and environmental prediction, rebuilding our capabilities when we still have the resources to be the best and give the American people state-of-science forecasts?   I hope so.


19 comments:

Wargens Heisergnarr said...

Nonsense, Europe has 500 million people hence much larger talent pool. Furthermore, France, UK (second best model), Germany all have their own global model so you cannot say that resources are spent much more efficiently.

Eric Blair said...

Around ten years ago I was watching the various forecasts concerning a possible snowstorm that could be coming down the pike to the Chicago area. Even back then, I had grown to distrust all of the cited sources except for the European model. With less than 48 hours to go, the laggards had predicted a few inches to drop, while the Euro model had increased it's forecast to a major snowstorm. The result? An enormous storm that dumped over twenty inches in less than 12 hours in the city proper, leaving hundreds of motorists stranded on Lake Shore Drive. The city was caught unprepared, it took almost a week to dig out the city, something that rarely happens in the winter. Never again.

Seattle Jim said...

As I am sure you are well aware, this is not a new problem. The National Research Council published a nice review of a major component of this problem in 2000 titled "From Research to Operations in Weather Satellites and Numerical Weather Prediction: Crossing the Valley of Death." I spent over 40 years working in the space weather world and saw this problem close up and personal the entire time. I am afraid this is a problem that is beyond the capability, or perhaps the will, of the two main players, funding agencies on the government side and researchers on the academic side, to resolve.

Peter Neilley said...

Cliff, You have articulated the problem well. Implicit in your blog, but I'll state it here explicitly, is that the success of the ECMWF is arguably the best example of what can happen when the research community is deeply embedded in and working with the operational community towards forecasting superiority. Your proposed solution is a start towards that for the US, including adopting a Boulder-based center. However, solutions confined to NOAA are insufficient and other agencies must be included, even though it is a much heavier lift as you point out. At a minimum, the National Science Foundation has to adopt the goal of achieving national forecasting superiority, including deep alignment and coordination with NOAA. Otherwise, NCAR and a lot of the NSF-funded academic community will not be sufficiently motivated to work on a holistic national goal, hence perpetuating the limited coordination that exists today between the research and operational communities.

J said...

I would love to see this blog post get national attention and see things change.

This is a list I quickly put together on how to submit tips to news organizations.

ABC: While many ABC affiliates allow tips via phone, Tweet, Test and email, the main ABC organization only accepts a web form submission. Google “abc news tip” and choose the “ABC News Tip Line – ABC News” link.

Associated Press: Only allows anonymous news tips. Android/Apple apps - Signal (+1-202-556-1927) and WhatsApp (+1-202-556-1927). There are also instructions on how to submit via Secure Drop. Google “AP News Tips” and choose the AP News Tips – Associated Press link.

BBC: Android/Apple app - WhatsApp (+44 7555 173285). Text a tip to 61124. Note: I have no idea if these would cost more to send from US devices. Email – haveyoursay@bbc.co.uk.
CNN: Tweet a tip to @TeamCNN. Text a tip to 772937. Phone – 404-827-1500 option 1.

FoxNews: Tweet a tip to @foxnews. Email – foxnewstips@foxnews.com. Phone – 1-888-369-4762.

NBC News Group: Android/Apple apps - Signal (+1-646-858-9310), Telegram (+1-646-858-9310) and WhatsApp (+1-646-858-9310). Email – tips@nbcuni.com.

JeffB said...

Don’t forget that many of these governmental organizations have also become highly politicized and often focus too much of their resources on climate change advocacy. You should contact the President. He seems eager to slash and burn the inefficiency and bloat of our federal government.

Eric Lenning said...

Me Blair, what was the date of the ‘failed’ Chicago snow forecast? Not the Groundhog’s Day Blizzard in 2011. That had a 48-hour lead time Blizzard Watch from the NWS. The city was extremely prepared, with one section of LSD blocked by a stalled bus but the rest of the roads around town virtually empty.

Alex said...

Make the Climate Great Again.

FixedCarbon said...

JeffB. Details please on "many of these governmental organizations have also become highly politicized and often focus too much of their resources on climate change advocacy.."

FixedCarbon said...

Fascinating article in the NYT.
https://www.nytimes.com/interactive/2018/08/01/magazine/climate-change-losing-earth.html

Unknown said...

Why is wx research a matter of us having "competitors" rather than there being worldwide cooperation?

TW B said...

Europe seems to handle government sponsored technology better than we do be it weather, nuclear power, trains, rapid transit, wind energy, and hydroponic farming to name a few.
One suggestion would be to award bonuses to the senior managers based on how well the models have worked by polling working level meteorologists (including Dr. Mass). This would be in lieu of them awarding themselves bonuses based on their own assessment which seems to be the norm in the federal government. Might work wonders.

John K. said...

Unknown - I've wondered the same thing. Why do we Americans think absolutely everything must be handled as a competition, with us as the "winner"? In many cases, like this one for example, that mentality does us more harm than good. I think the more mature cultures of the world have moved past that.

stevemcn said...

Any thoughts on the nomination of a meteorologist, Dr Kevin Droegemeier, to White House Office of Science and Technology Policy? Seems related to this topic.

jimijr said...

"Where there's a will, darlin', sure 'nuf there's a way." I agree with Mr. Neilley, don't give it all to NOAA, they're all oceans, you know that. Thanks for bringing this issue forward.

Quit yer whinin'! said...

Dr. Mass has written about this phenomenon before, and I’m sure he could elaborate again.

Foo said...

@TW B Performance based incentives sound like a good idea, and there are likely to be ways to do that using quantitative and completely objective criteria.

Of course there need to be common sense limits to that approach - for example, it's not a good way to assess some types of basic research or experimental work far removed from operational uses but still very important.

Foo said...

@John K I think many people - myself among them - recognize the difference between constructive vs. destructive competition. For an example of the former, consider the world of sports: Would athletes continue to improve and to break records without competition to motivate them and spur them on?

How about business, industry and technology: Would businesses innovate and deliver better products at lower costs if competitors didn't make these achievements necessary? Is it telling that the opposite of healthy economic competition isn't harmony and progress, but rather monopoly?

Would the United States have accomplished its moon landings - perhaps our species defining scientific feat, even to this day - without the heat of a geopolitical rivalry to keep us focused and determined to succeed?

Have you ever met anyone who achieved a successful life without occasionally setting out to do something faster, better and smarter than his or her rivals?

It's only human to benchmark oneself against others and to draw motivation from the desire to be recognized as the very best - at least if one considers an activity to be truly important. When you harness that desire (and good luck trying to suppress it - you may not like the unintended consequences) then great things are possible.

Of course, how to define "constructive" competition is the big question. Some key elements come to mind - for example, recognizing that your competitors deserve the same respect and recognition for their achievements as you expect for yours. Another might be to reject the idea that competition in one arena requires the competitors to play a zero-sum game anywhere else they meet, as well. Yet another might be to agree that in many cases, the point of "winning' is to have the privilege of sharing an achievement with everyone else - i.e. medical advances, basic scientific knowledge.

Or maybe it's as simple as doing everything you can to beat the hell out of your rivals when you hear the starting gun - but also looking forward to meeting them all for a drink when the race is over.

It just seems to me that it's more useful to turn competition to everyone's advantage than it is to insist that competition is always a zero-sum game and that it needs to go away. It's not going anywhere, and history is full of examples why that CAN be a very, very good thing if it's handled correctly. And I think that's just as true when it comes to the big things - the things we're talking about here - as it is of the small things.

I can't think of a better way to get more powerful and accurate models than encouraging competing ideas and approaches. But then again, I also can't think of a reason why something this important and useful to saving lives and property wouldn't be shared - think in terms of Open Source software - once we've sorted out the winning approach.

Or to boil things down one more time: It's not about whether we compete. It's about how we compete.

Unknown said...

You cannot possibly manage all that diverse talent that easily.