Usability Testing of Drought Forecast Visualizations for Improves Understanding and Decision Making

Forecast visualizations and graphics are widely used to communicate important, often complicated scientific and risk information. These visual data products are used across multiple sectors to make key decisions related to managing or responding to natural hazards such as drought. However, empirical research has revealed that: 1) the user base and audience segments for these products are often undefined or poorly defined; and 2) these products have largely been designed in a social scientific void with technical information being prioritized over the presentation and design of visual communication to aid effective decision making. This has led to the proliferation of forecast visualizations that are not intuitively designed, are difficult to interpret, and are inaccessible to non-expert general audiences. To overcome these usability and understandability challenges in decision support products, we applied a multi-pronged framework including diagnosing, evaluating, testing, and improving visual risk communication. Demonstrating the applicability and usefulness of this methodology for integrating social science research findings into best practices for designing effective drought risk communication, this study focused on the monthly and seasonal deterministic U.S. Drought Outlooks produced by NOAA’s Climate Prediction Center (CPC).

We worked with NOAA CPC to identify key message objectives and decision support goals for the U.S. Drought Outlooks, following previous successful research testing, redesigning, and operationally improving NOAA CPC’s temperature and precipitation outlooks. Based on our extensive review of multidisciplinary literature, design priorities identified by NOAA CPC and the National Integrated Drought Information System (NIDIS), and previous empirical findings, we proposed modifications to the legend category labels, color schemes, and legend organization of the current US Drought Outlooks. We then designed an online survey-experiment for a control-treatment test to demonstrate the impacts of both the existing version and our proposed redesigned versions of the drought outlook graphic on user preference, intuitiveness, usability, and understandability. Our findings demonstrate the extent to which our redesigned graphics - if operationalized - would strategically and statistically improve the usability and understandability of a vital drought risk communication and decision support product.

Our sample of 800 U.S. adults over the age of 21 largely represents college-educated, working age individuals and reflects U.S. census data in its gender, race, and ethnic diversity. We chose to over-sample college-educated, working-age adults because previous research and interactions with NOAA CPC indicated that these characteristics represent most real-world end users of the U.S. Drought Outlooks. All participants answered initial survey questions assessing their ability to correctly interpret the drought categories in the U.S. Drought Monitor (USDM) - which are the underlying categories that the NOAA CPC monthly and seasonal U.S. Drought Outlooks are based on. Participants were then randomly assigned into one of four groups - a control group which included questions about the current U.S. Drought Outlook graphic, and three treatment groups, each representing a redesigned graphic we were testing. Each of the redesigned graphics representing the three treatment conditions had different color schemes but the same forecast categories in their legend. Participants in each group only saw one kind of graphic - either the control or one of the three treatments.

Results reveal that most participants were confused about and incorrectly perceived the USDM categories of severe, extreme, and exceptional drought. This has direct implications for how accurately users of the U.S. Drought Outlooks are able to differentiate between drought forecast categories, thereby influencing the decisions they make in response to this risk information. We also found that our modified drought forecast legend categories were correctly understood by most participants. For example, most participants (85%) were able to correctly determine that if drought conditions were forecasted to go from moderate (D1) to extreme (D3), according to our modified legend, that would mean that drought would worsen. When evaluating whether participants associated individual colors in our modified color schemes to the appropriate drought forecast category, we found that most participants correctly assigned colors from all three different color schemes, each representing one of the three experimental treatment conditions, to their associated drought category. For example, participants largely assigned a darker sand brown color to the ‘drought worsens’ category, and a lighter sand color to the ‘drought develops’ category. Conversations with NOAA CPC revealed that the intended key message of the U.S. Drought Outlooks is showing the change in drought from one forecast period to the next, as opposed to the intensity of drought forecast which is a common misperception with these products. When asked what they thought the key message of the graphic they saw was, more than half the participants in each of the three treatment groups correctly answered ‘change in drought’. However, an overwhelming 78.5% of participants in the control group thought the existing drought outlook graphic shows intensity of drought forecasted. Notably, this means that even in an educated sample of participants, the key message of these forecast visualizations is largely misunderstood.

Overall, compared to the existing U.S. Drought Outlook graphic, we found that all three of our proposed redesigned graphics were more preferred by users. Pending upcoming causal analyses, we will be able to demonstrate the extent to which these likeability results also hold up in terms of understandability, providing evidence of whether the participants interpret the redesigned graphics more accurately compared to the current drought outlook visualization. The modified graphics were designed to be Section 508 compliant so that people with color vision deficiencies could visually distinguish between the different colors used to represent drought conditions. Initial results indicate that using our modified graphics did not require a great deal of mental effort, and that participants largely felt these redesigned graphics were clean and uncluttered in their design, and felt confident in their ability to interpret them.

This study will support NOAA and NIDIS in using evidence-based decisions to make operational design changes to the U.S. Drought Outlooks that will improve their usability and understandability, while also underscoring the importance of integrating social scientific research and best practices into the design of other forecast visualizations and decision support products.