An independent evaluation of the OK-FIRST decision-support system

In 1996, the Oklahoma Climatological Survey embarked upon a project to support emergency managers in the use of weather information for severe storm and related operations. The OK-FIRST Project provides emergency managers and law enforcement officials in Oklahoma with access to real-time radar and surface weather observations and links to National Weather Service Products. Prior to gaining access to the data, each emergency manager must complete a training course consisting of (1) a one-week workshop on use of computers and relevant software, and (2) a second one-week workshop on interpretation of data, especially radar data.

An important part of determining how well-equipped these emergency managers were to use the new information was a formal independent evaluation. The evaluation consisted of three components. First, an initial survey of participants established a baseline of their prior use of various types of weather information, characteristics of their local communities and official duties, and their length of service. Second, at each workshop, participants were given a pre-test and a post-test to establish what they knew coming into the workshops and what they knew at the end. Third, follow-up focus groups and interviews with participants provided an assessment of how the information was used in operational settings.

Overall test scores showed a statistically significant improvement in both computer training and data interpretation as a direct result of the workshops. In addition, the overwhelming majority of participants self-reported that they found all aspects of the training and data acquisition as very or somewhat useful. NEXRAD information was consistently listed as the most useful product, while hydrological products were consistently listed as less useful. The data collected during the evaluation of OK-FIRST indicate that the OK-FIRST team has been successful in meeting the key needs of program participants and has empowered local officials to make decisions based on up-to-the-minute information. The evaluation identified an unanticipated outcome: participants were able to extend their involvement from emergency situations to a wide range of government and public service functions

The evaluation was useful not just in assessing changes in participants knowledge, but it provided immediate feedback to the OK-FIRST staff so that they could correct areas in which participants were having difficulty and better design the workshops to meet the needs of the participants. Close cooperation between the project and evaluation staffs helped to identify and resolve these deficiencies. A critically important lesson learned from the OK-FIRST project is that the integration and coordination of training, access to real-time data, and ongoing support are vital to the success of the program.