June 14, 2012

Weather-X

There has been a lot of press coverage of late about the successful commercial space venture Space-X.   This private sector effort was able to build a launch vehicle (Falcon-9) and a spacecraft (Dragon) that not only worked flawlessly, but made a delivery to the international space station.  With NASA out of the manned space business for a while with the end of the Space Shuttle program, this feisty young company has taken on a role previously held only by a U.S. government agency, and they did it far faster and cheaper than a Federal entity could ever do.   To its credit, NASA has been supportive of Space-X's activities.

Falcon-9



Dragon

So now let's consider weather prediction.  A portion of U.S. numerical weather prediction has been done outside of the National Weather Service/NOAA and the U.S Navy (Fleet Numerical):  in the private sector and academic institutions (like the UW!).  However, the core global prediction infrastructure for the U.S. has really been a NWS function and most of the non-governmental efforts are dependent on NWS global grids, data assimilation, and other products.

Unfortunately, the current situation is not good.  The National Weather Service prediction efforts are crippled by inadequate computer infrastructure, lack of funds for research and development, an awkward and ineffective research lab structure out of control of NWS leaders, and government personnel rules that don't allow the NWS to replace ineffective research and development staff.  Lately there has been talk of furloughs for NWS personnel and a number of the NWS leadership are leaving.  The NWS has fallen seriously behind its competitors (e.g., the European Center for Medium Range Weather Forecasting, UKMET office, Canadian Meteorological Center) even though the U.S. has a huge advantage in intellectual capital (U.S. universities and the National Center for Atmospheric Research are world leaders in field, as are several U.S. government research labs--e.g, NRL Monterey).

So we have a U.S. government entity that has lost leadership in a key technological field of huge importance for the nation.  It no longer has the resources to be state-of-the-art and is hemmed in by ponderous governmental regulations.   Sounds quite a bit like NASA's situation, doesn't it?   And might the solution be the same?    Could a private company develop the capability for state of the art global prediction, high resolution regional prediction, and the ability to move into the probabilistic prediction we know represents the future?  Is it time for Weather-X?

The clear answer ... you bet.

Let me be honest, I am really surprised that a private sector firm hasn't taken on this challenge already, considering the obvious potential to create a forecast entity that could produce a product that would be in considerable demand.  For example, U.S. companies are spending millions of dollars to get the European Center (ECMWF) forecast model output---and it is possible to do far better than the ECMWF. 

To create this new firm one would need large computer resources (.5 to 1 petaflops would be a good place to start).   That would cost 10-15 million dollars to buy from scratch, but many companies (e.g., Microsoft, Google, Boeing, Amazon, IBM, major defense contractors, and more) have it already.

The new firm would need computer models, but those are already freely available, and research folks like myself could easily be bribed to help with some research grants (trust me on this!).  In fact, the research community would be lining up at the door to help if some $ was available.


The most difficult aspect is the data assimilation part... securing all the satellite and observational assets needed to initialize the forecasts---but many of those are in the public domain and I suspect that some deals could be made with NASA, NOAA, and the European Space Agency.  And there are some potentially very useful data sets that the NWS can't afford today that would be a substantial value (e.g., weather data from commuter planes).

Yes, it would probably take 5-25 million dollars to get started on this, but consider that the folks at Space-X invested 100 million dollars for a venture that was far more speculative.   Local investors are willing to spend as much or more ON A BASKETBALL TEAM. 

There is no doubt this idea would work. The best weather forecasts in the world would be a valuable commodity for many industries that would be ready to pay (e.g., renewable energy, agriculture, power generation and distribution, shipping...the list is endless).   Want to make your search engine attractive?   Have the best forecasts available on it!

Some of you might argue that my colleagues in the NWS might not be pleased about such an upstart.  But if NASA can happily agree to work with a private sector firm to take on some of its work, why should the NWS be any different?  I suspect the private sector could fulfill this function more effectively and at less cost.  And the benefit to the nation of vastly improved weather prediction guidance could be enormous.

As noted above, there are several companies that already have the computer and IT infrastructure in place to take this on (such as Microsoft, Google, Apple, and Amazon).  Several of them are found here in the Northwest.  All that is missing is the vision to see the opportunity and seize it.







10 comments:

  1. The main concern I'd have about privatizing NWP is the hindering of science. If a private company improves a NWP model or some data assimilation method, will it share the results with the scientific community? Not likely. Science can be crippled with privatization.

    Also, is Space-X really a stand alone private company, or are they leaning on the government? The answer is the latter. They are basically subcontracted by NASA.

    If Weather-X were to work, it would have to be in a tight partnership with the NWS. There are too many important aspects, such as data collection, which is a necessary first step in NWP. There is a lot more to it than simply running a model that you didn't even create in the first place. A lot of private firms are finding that out the hard way.

    ReplyDelete
  2. Dr. Mass, very intriguing blog entry. I would love to see something like Weather-X come into existence. I agree with "Unknown" when he says that we must keep things open and transparent to the scientific community.

    If we can't find the money, perhaps we can involve large companies into the investment. Insurance companies jumps to my mind, and they would offer quite a bit if their loses could be minimized through better model output.

    ReplyDelete
  3. Unknown and Wright,
    I agree completely...the science needs to be completely in the open and in the published literature. I don't see any issue with this...in fact, the technology development will be far more rapid if done in the open...to the advantage of Weather-X. There are many fields in which technological services or products are sold and science behind it are in the public literature. And of course there are patents....universities love patents, if they can make some money off of it. Right now there is no way for university innovations in weather prediction to be monetized....weather-x might change that...cliff

    ReplyDelete
  4. Cliff,

    I am not sure if you have looked at The Climate Corp. it is a good example of a private entity doing work in this area. A friend of mine met with them and was very impressed with the way they are dealing with their data and computation needs.

    http://www.geekwire.com/2012/climate-corp-picks-seattle-office-citing-proximity-amazon-web-services/

    Erick

    ReplyDelete
  5. For the cost of bringing an NBA team back to Seattle, we could have a better-than World Class weather prediction system!?! No brainer!

    ReplyDelete
  6. Expect Paypal links to Space X as Musk is founder of both. I am not sure how you make the business case for the wider market of general users. If the Weather Channel was a pay as you go concern instead something that just shows up on cable TV I don't think they would be here today. We expect our weather information to be free. That said I to am surprised that Google or Apple have not done more in this area. Maybe they figure out it is a loser financially. Cliff can attest how hard it has been for his effort to garner recurring funding from the agencies that depend on it.

    ReplyDelete
  7. Professor Mass, I have to play devil's advocate a little. Why does the world need several competing centers all preparing a global weather model? Couldn't the U.S. just contribute to the European Center's model, collect our data to report, and prepare local and regional forecasts?

    ReplyDelete
  8. Cliff, I've noticed that already this year there have been two hurricanes in the Eastern Pacific off Western Mexico and two typhoons in the Western Pacific. That seems like abnormally active weather very early. One of the typhoons and one hurricane were before the "official" start of the season.

    Is that simply within historical range and lacking significance?

    Thanks

    ReplyDelete
  9. Unknown: Also, is Space-X really a stand alone private company, or are they leaning on the government? The answer is the latter. They are basically subcontracted by NASA.

    As well, the technology package employed by SpaceX is almost entirely based on research and development performed by the government.

    Not to diminish the SpaceX achievement one iota but a lot of folks are pointing at the recent launch and using it as an example of failure of governement. Hardly the case; SpaceX is the tip of a pyramid of effort on the part of taxpayers. Hats off to us, too.

    Cliff, off-topic but seems to me that Dallas' recent $1 billion-plus property damage due to hail is possibly a case of the intersection of weather and climate? Forecasters seem to have been caught by surprise at the rapid intensification of the supercells dropping this hail on Dallas. Not to call this out as a specific case of climate-influenced intense weather but I'm wondering about the continuum between weather and climate models. How do the two talk to one another, if at all?

    ReplyDelete
  10. Some interesting info/perspective concerning NWS, via Robert Grumbine


    "NOAA/National Weather Service (NWS) budget issues and possible furloughs: (again: I don't speak for my employer, whoever that may be)

    I confess that I do know Jack Hayes, the now-retired head of the National Weather Service personally and professionally. How this came about is something of a story, which I won't be telling. Suffice it to say that I do know him, and think well of him from my modest first-hand experience.

    The recent news about improper redirection of funding within the NWS is interesting.

    The impropriety is not one of personal gain. Whoever did so does not seem to have profited by the redirection.

    The direction of impropriety was to spend more for getting weather forecasts and weather warning delivered to the US public than Congress authorized.

    Carrying out Congress's direction would have meant either all weather forecast offices issuing fewer forecasts/providing less guidance for protection of life and property, or closing some forecast offices outright.

    Since it's Congress's job to make that decision, it should be followed. If citizens object, they have a ballot box.

    The total National Weather Service budget for 2012 is less than 1 Big Mac per person, per year."

    ReplyDelete

Please make sure your comments are civil. Name calling and personal attacks are not appropriate.

The Real Story Behind Tuesday's Windstorm and Massive Power Outage

 If you read media reports, such as in the Seattle Times, one is told that the localized winds on Tuesday evening were the result of an unus...