“Big Data Assimilation” Revolutionizing Severe Weather Forecasting (Core Science Lecture)

In 2013, the Japanese government started a strategic funding program for the Big Data science on fundamental technological developments and big data applications. One of the two proposals awarded in the field of big data application was “Big Data Assimilation” for super-rapid 30-second cycle of an ensemble Kalman filter aiming at local severe weather forecasting at a 100-m resolution with 30-minute lead time. This numerical weather prediction (NWP) system is truly revolutionary, so that it gives us 30-minute forecasts at a 100-m resolution refreshed every 30 seconds, 120 times more rapid than the current hourly-updated systems. With this system, we could be prepared for sudden local torrential rainfalls that may cause flash flood and river outflow only within 10-20 minutes.

This revolutionary NWP is only possible due to the most recent developments on advanced sensing and computing technologies. The most recent Phased Array Weather Radar (PAWR) can make a three-dimensional volume scan up to the 60-km range only in 10-30 seconds at 100-m radial and 300 azimuthal resolutions with 100 elevation angles. Also, the Japan Meteorological Agency's next-generation geostationary satellite Himawari-8 has a capability of the super-rapid scan every 30 seconds for a limited region. These sub-minute data may be frequent enough to capture the nearly linear evolution of convective activities whose typical lifecycle is less than an hour. Recent NWP models can simulate the individual convective cells at a 100-m or higher resolution. Namely, assimilating these 30-second data into NWP models may lead to accurate representations of the lifecycles of each convective cell. However, these new observing platforms provide two orders of magnitude more data, and an effective use of these so-called Big Data in very short range NWP is a real challenge.

We estimated the computational amount to be 4 Peta floating-point operations for a single step of the super-rapid 30-second cycle. Using the Japanese 10-Peta-FLOPS K computer, it takes about 4 seconds if we assume relatively optimistic 10 % computational efficiency. If we could transfer the 200-GB NWP ensemble data and 2-GB observational data within 10 seconds, we could apply this super-rapid NWP system in real time. However, these estimates are still preliminary, and our recent experiments using the K computer suggest more demands.

This presentation will discuss the concept and recent progress of the pioneering “Big Data Assimilation” research.