A Comprehensive Dataset to Study and Predict Weather Impacts on Aviation

To study weather impacts on aviation, a fusion of aviation and meteorological data is required. Within the domains of aviation and meteorology, a huge volume of data is collected and archived from multiple sources in disparate forms (structured, unstructured). As a part of an ongoing study, a data pipeline has been developed, using state-of-the-art big data technologies, to prepare a large-scale dataset of spatio-temporally correlated flight, airport, and weather records [1]. The hardware required for this study was procured through an NSF-funded Unidata Hardware Equipment Award in 2023 [2]. The team intends to share this prepared comprehensive dataset with the Unidata community[1] to foster the study of weather impacts on airport and airspace operations. It is expected that the dataset will benefit both aviation and meteorology communities, especially those studying the weather impacts on aviation, as the system will accumulate data over time and future data mining opportunities will only increase.

At Embry-Riddle Aeronautical University (ERAU), the NextGen ERAU Advanced Research (NEAR) Laboratory has collected and logged an enormous volume of National Airspace System (NAS) data over the years. Every 24 hours, 25 GB of compressed data are logged, which amounts to 290 GB of uncompressed, XML-formatted data files. An estimated 105 Terabytes are logged annually.

The Meteorology Program within the Department of Applied Aviation Sciences (AAS) at ERAU receives and archives weather data utilizing several National Oceanic and Atmospheric Administration (NOAA) sources, as well as academic providers. For example, considering NOAA Global Forecast System (GFS), the typical volume from one forecast exceeds 100 GB and arrives four times per day, for a total daily volume exceeding 425 GB. A month of data exceeds 12 TB, and a year exceeds 155 TB.

Traffic Flow Management System (TFMS) [3] is one of the datasets collected at the NEAR lab and is used by the air traffic management to exchange near real-time flight information and traffic flow management data within the NAS users (internal and external). Relevant TFMS messages include flight data and flow data. The flight data includes information about flight plans, flight tracks, departure and arrival time notifications, boundary crossings, flight cancellations, etc. The TFMS flow data includes information about Traffic Management Initiatives (TMIs) including reroutes, ground stops (GS), and Ground Delay Programs (GDP).

A data pipeline has been developed by the team using state-of-the-art big data technologies including Apache NiFi, Hadoop Distributed File System (HDFS), and Apache Hive. Apache NiFi is used to ingest and process big data (TFMS, GFS, and METAR) from disparate data sources. Data acquired from multiple sources are stored and archived in a distributed file system. The data is analyzed and used to perform machine learning tasks and knowledge extraction.

Benefits

The dataset, which the authors intended to share once complete, contains comprehensive information about all flights that were impacted by one or more TMIs because of inclement weather conditions at or around the departure airport, or at the destination airport. This includes activation of GDP and GS at the airport due to bad weather conditions (including but not limited to wind, low ceiling, thunderstorms, snow/ice, etc.), airspace flow program, flight rerouting, and deicing operations at the airport. The airport configuration (APTC) messages of the TFMS capture the weather conditions at the airport, which includes weather conditions like Instrument meteorological conditions (IMC), Low IMC (LIMC), visual meteorological conditions (VMC), and Low VMC (LVMC). Additionally, for detailed weather conditions at the airports, weather observations from the ASOS data that provide the ground truth have been merged with the TFMS dataset. Further, the forecasts from the Global Forecast System (GFS) are to be added to this dataset. To the best of our knowledge, such a dataset is not available publicly.

Literature shows that using machine learning to predict flight delays has been studied widely. However, the most widely used dataset in these studies is being received from the Bureau of Transportation Statistics (BTS), which makes available a post processed record for airlines participating in BTS data collection. The data are reported by the airlines, and the assignment of weather impacts is not always accurate. The researchers don’t have a real-time or near real-time access to this dataset. For example, at the time of this submission (August 2024) the latest data provided by the BTS is April 2024. On the other hand, ERAU has access to the TFMS data as it is recorded, and TFMS provides near real-time flight, flow information of the airspace, and weather conditions at the airports. The TFMS dataset is expected to be significantly more helpful to researchers studying airport and airspace operations. The availability of such a dataset is expected to facilitate research in areas like predicting the nature and duration of TMIs, and consequently flight delays, delay propagation in the airspace and airport operations, correlations in such decisions with observations and forecasts, and other yet to be conceived questions that could be answered.

References

[1] “Joint Session J8A – Cloud Computing for Big Data in Atmosphere, Ocean, and Climate”. https://ams.confex.com/ams/103ANNUAL/meetingapp.cgi/Session/62151

[2] “Unidata 2023 Awards”. https://www.unidata.ucar.edu/community/equipaward/index.html#2023

[3] “Traffic Flow Management System (TFMS)”. https://aspm.faa.gov/aspmhelp/index/Traffic_Flow_Management_System_(TFMS).html#:~:text=TFMS%20(previously%20ETMS)%20is%20a,and%20departure%20and%20arrival%20messages.

[1] https://www.unidata.ucar.edu/projects/#idd