Applying Statistical Decision Theory to a Field Experiment with Multiple Research Objectives: The DC3 Campaign

In airborne field campaigns, investigators confront complex decision challenges concerning when and where to deploy aircraft to meet scientific objectives within constraints of time and budgeted flight hours. An automated flight decision recommendation system was developed to assist investigators leading the Deep Convective Clouds and Chemistry (DC3) campaign in spring—summer 2012. In making flight decisions, DC3 investigators needed to integrate two distinct, potentially competing objectives: to maximize the total harvest of data collected, and also to maintain an approximate balance of data collected from each of three geographic study regions. Choices needed to satisfy several constraint conditions including, most prominently, a limit on the total number of flight hours, and a bound on the number of calendar days in the field. An automated recommendation system was developed by translating these objectives and bounds into a formal problem of constrained optimization. In this formalization, a key step involved the mathematical representation of investigators' scientific preferences over the set of possible data collection outcomes. Competing objectives were integrated into a single metric by means of a utility function, which served to quantify the value of alternative data portfolios. Flight recommendations were generated to maximize the expected utility of each daily decision, conditioned on that day's forecast. A calibrated forecast probability of flight success in each study region was generated according to a forecasting system trained on numerical weather prediction model output, as well as expected climatological probability of flight success on future days. System performance was evaluated by comparing the data yielded by the actual DC3 campaign, compared with the yield that would have been realized had the algorithmic recommendations been followed. It was found that the algorithmic system would have achieved 19%—59% greater utility than the decisions undertaken by DC3 investigators, depending on the scoring method used.