Towards a Community Resource for High-Volume Model Data Processing Near NCEP

Towards a Community Resource for High-Volume Model Data Processing Near NCEP

Peter P Neilley, The Weather Company, Atlanta GA

Ben Kyger, NCEP Central Operations, NOAA, College Park MD

Mohan Ramamurthy, Unidata Program Center , UCAR, Boulder CO

The availability of robust and skillful weather and climate model forecasts in real-time from the National Weather Service has been instrumental in helping to create a vibrant weather enterprise in this nation supporting academic, commercial, and numerous other interests and activities. The general paradigm is NCEP creates the model data and then distributes the data to end users for display, analysis and various other value-added processing. However, practical issues regarding the transport of the model data from the supercomputing centers to end-users limit the scope and timeliness of the data that can be delivered to the enterprise via this paradigm. Bandwidth constraints of NOAAPort and the internet are the primary limitations. As a result, model data must be subsampled or filtered before distribution to end users and hence the optimal value of the NCEP model data suite to the nation has not yet been realized. For example, model output is available typically on 1 or 3 hour intervals, far less frequent than the maximum possible output of every time step. The Global Ensemble Forecast System (GEFS) data is available only on standard pressure levels rather than on the (higher resolution) native vertical coordinate. More complete model data availability to the enterprise would inspire novel new uses of the model data and enable a richer interaction between NCEP and the enterprise in evaluating and improving our nation's modeling capability. The issue of limited bandwidth of transporting model data to end users will be compounded in the future as increasing computing power (and hence the ability to create model data) outpaces the growth in affordable communication means. One way for the enterprise to more fully exploit the model data is to enable community model data processing to occur at or near where the model data is created, thereby alleviating the need to transport the data to the user's facility first. This paradigm shift of “bringing the processing to the model” rather than “bringing the model data to the processing” is the seminal idea in the Open Weather and Climate Services (Neilley et al., 2011) concept developed by the Environmental Information Services Working Group and recently endorsed by NOAA.

In order to advance this notion, a time-limited, experimental data processing center in the same building as the new NCEP supercomputing facility in Reston, VA is being created. This experimental data center will connect directly to the NCEP supercomputer file system via a high-speed fiber cable. This connection will enable mirroring of the more complete model data set within the experimental data center thereby making available to applications running at the data center a more complete and timely set of the model data. Computers supplied by participating community members and installed at the experimental data center will then be able to connect directly to the mirrored data file system, allowing value-added processing to occur and the (lower volume) derived results to be shipped to participant's home base or other useful destinations.

The main objectives of the experimental data center is to perform basic, prototype activities that emulate and exercise what a more full-scale, robust implementation of the concept may look like. It will help inform and/or overcome many of the the legal, security, technical and procedural challenges that are raised by this concept. Although no concrete, specific architecture for a final operational state of a system has been formed, we envision eventually enabling a commercial, “cloud-style” open service in which computing services can be purchased or leased by enterprise members, with access to the full NCEP model data being included as part of those services. The presentation will include an update on the status of the experimental data center and an invitation to the community to participate in the experiment.