Combining the CASA radar observations together allows for the generation of meteorological products beyond what is typically associated with radar. To achieve this, the ability to move voluminous data quickly to centralized processing locations and to utilize computing resources in an efficient manner are very important considerations. In fact, it is precisely when the system is most important to the public, when tornadoes, wind events, and flooding are ongoing, that computational demands will be highest, and network reliability will be lowest. These instances are relatively rare, so we seek out an infrastructure where resources can be acquired on demand, with a national footprint to quickly move processing out of harm's way. For this purpose, the virtual lab provided by the Global Environment for Network Innovation (GENI) has been a natural fit.
There are several aspects of GENI that are used:
1) The elasticity of resources. Overall CPU utilization is a function of the weather phenomena contained in the radar data, the subset of meteorological products being generated, and the geographic size and resolution of the network domain. Rather than dedicate computers to the worst case, (widespread severe weather, all of our products produced at their highest resolution) we have implemented software to acquire and release virtual compute instances dynamically, triggered by a suite of meteorological detection algorithms. Multiple instantiations of the same product can be produced with different parameterizations or operating on different subsets of data.
2) Network commonality. Several of the radars in the DFW network and the standard, centralized radar operations center, have their network provided by the Lonestar Educational and Research Network (TX-LEARN). Fortunately, Texas A&M University and the University of Houston are also provisioned by LEARN and host GENI racks on which we can acquire virtual machine instances. Throughput within the LEARN network is quite high, latency quite low. This allows us to transport radar data in near real time to the VMs for quick processing. Timely analysis is crucial for stakeholders, and the ability to send out radar moment data and derivations thereof with very little delay at data rates of >100mbps provides a significant advantage over traditional commodity internet.
3) Layer 2 network connectivity between GENI nodes. Layer 2 networks are extremely fast and allow for even greater flexibility in the choice of compute locations by reducing transport delays. For example, there are large clusters available for distributed processing at the University of MA Amherst that would otherwise be inaccessible due to the time constraints and the delay in sending voluminous data across the country. However, with the layer 2 connection, we can achieve rates an order of magnitude higher than with standard internet2 links. The huge throughput gains allow for true radar signal processing to be done in controlled facilities rather than in the field.
4) Non-Real Time Processes. We utilize the GENI resources for cost effective post-analysis and case study data distribution as well as experimental real time development all without having to impact operations of the running system. When strong severe weather hits there is immense research and end-user interest thereafter. Resources can be strained by data downloads, post processing algorithms, and experimental detections. Similarly, before deployment in the real time operational system, algorithms must be tested and benchmarked in a realistic manner, but without risking operations.
GENI has been an effective tool for these purposes. We describe lessons and the potential for further use herein.
- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner