From Data to Decision: Exploring User Engagement with Machine Learning Models During Cold-Stunning Events

Cold-stunning events, when water temperatures drop to threatening levels for wildlife (e.g., endangered sea turtles), pose a substantial risk to coastal systems along the Texas coast. Mitigating these risks requires having useful tools for predicting water temperatures and making decisions to protect wildlife, which involves significant economic and transportation disruptions. The Texas Marine Coldwater Response Collaborative (TCRC) was established to manage these risks and decision-making. An artificial intelligence / machine learning (hereafter, “ML”) model was developed to assist their risk assessment and decision-making. The research reported here aims to (a) understand decision-making landscapes, informational needs, and communication pathways of our users within the TCRC and (b) co-develop useful representations of ML-derived uncertainty of water temperature predictions more specifically pertaining to the timing of cold-stunning events.

The co-development process is conducted with the TCRC through continued relationship-building and stakeholder engagement. This information will be collected through structured interviews conducted with a small subset of TCRC members, focusing on members’ background and job responsibilities, experiences and decision-making processes leading up to, during, and proceeding cold-stunning events, their interpretations and trust in current operational cold-stunning guidance, and their thoughts and understanding of preliminary uncertainty representations of water temperature predictions derived from a ML ensemble model. Interviewees’ responses will be used to synthesize patterns observed in topics, ideas, and statements using a mix of qualitative and quantitative analyses.

We create an inventory of networked individuals and groups within the diverse base of the TCRC including who and/or what groups are involved during certain stages of cold-stunning events and certain decision-making processes when using the ML-derived information. We also synthesize what information is used for critical decisions within the TCRC for adequate cold-stunning advisory, their perceptions of hypothermic threats throughout various stages of the event and their understanding of uncertainty water temperature information. This research will also result in the identification of the contextual factors that contribute to TCRC members’ trust in the current operational ML cold-stunning model and associated advisory information.

An improved and trustworthy model that provides uncertainty estimates of water temperature predictions and the timing of threatening water temperature ranges is valuable for precautionary decision-making processes within the TCRC. This research will help gain a deeper understanding of how stakeholders work together to make key decisions with the ML predictions during different stages of the cold-stunning events and the dynamic nature of trust throughout the cold-stunning advisory process. Understanding how users' informational needs, communication pathways and risk perceptions change throughout different phases of the cold-stunning event as well as some of the contextual factors of TCRC members’ perceptions of trust in the cold-stunning model will help to determine what stakeholders need when creating new forms of uncertainty quantifications and how to optimize the utilization of this ML-derived information.