Evaluation of Decision Support Research and Application Within the Climate Services Enterprise
This presentation describes a framework for evaluating decision support research derived from process theory, rather than program theory that frames the logic model that is commonly used to evaluate problem-focused research; as recent reviews have emphasized, science for decision support to manage wicked problems is best handled as a process, in part because those problems are not solvable or even well-defined, and conditions can evolve rapidly. The framework consists of five elements: transformation, transfer, translation, process, and decision support systems. The first three elements are based on integrating existing evaluation hierarchies and criteria related to academic products calibrated across diverse disciplines, technological research and development innovation, and ways in which knowledge and decision support systems cross boundaries. The synthesis results in a framework that considers (1) the nominal level, scale, and scope of change produced in networks and organizations by research outputs, outcomes, or impacts, and (2) aspects of innovation, including the scope of the innovation's novelty, the level of scholarship and engagement involved, and the notion of sustained process.
The process element of the evaluation framework addresses the implementation of decision support research. It combines a wicked problem's contextual attributes, a set of recommended principles for decision support for dealing with wicked problems (NRC, 2009), a recommended decision process of deliberation with analysis (NRC, 2010), and a hierarchy of engagement with networks. Within the framework, evaluation of decision support systems builds on evolving standards of digital scholarship that prioritize context-appropriate application over documentation of theoretical groundings within academic literature or metrics that can be rendered irrelevant as technologies advance.