Evaluation of Decision Support Research and Application Within the Climate Services Enterprise

The climate service enterprise seeks to link science and society more effectively, emphasizing the connections of climate to other stressors and opportunities, across spatial and temporal domains, disciplines, organizational roles, and issue domains. Parts of the enterprise have long provided use-inspired or problem-focused science, reaching the public and decision makers through extensive outreach efforts and new products and tools. However, recent National Research Council (NRC) reports and the National Climate Assessment recognize that challenges remain in the evaluation of decision support research, especially related to “wicked problems”, to the increasing vulnerabilities of coupled human and natural systems where demands are nearing or exceeding supplies, and to widespread change driven by disruption of natural hydroclimatologic variability.

This presentation describes a framework for evaluating decision support research derived from process theory, rather than program theory that frames the logic model that is commonly used to evaluate problem-focused research; as recent reviews have emphasized, science for decision support to manage wicked problems is best handled as a process, in part because those problems are not solvable or even well-defined, and conditions can evolve rapidly. The framework consists of five elements: transformation, transfer, translation, process, and decision support systems. The first three elements are based on integrating existing evaluation hierarchies and criteria related to academic products calibrated across diverse disciplines, technological research and development innovation, and ways in which knowledge and decision support systems cross boundaries. The synthesis results in a framework that considers (1) the nominal level, scale, and scope of change produced in networks and organizations by research outputs, outcomes, or impacts, and (2) aspects of innovation, including the scope of the innovation's novelty, the level of scholarship and engagement involved, and the notion of sustained process.

The process element of the evaluation framework addresses the implementation of decision support research. It combines a wicked problem's contextual attributes, a set of recommended principles for decision support for dealing with wicked problems (NRC, 2009), a recommended decision process of deliberation with analysis (NRC, 2010), and a hierarchy of engagement with networks. Within the framework, evaluation of decision support systems builds on evolving standards of digital scholarship that prioritize context-appropriate application over documentation of theoretical groundings within academic literature or metrics that can be rendered irrelevant as technologies advance.