The verification of the weather model is done using a combination of proprietary and open source code, the latter being the NCAR Developmental Testbed Center (DTC) Model Evaluation Tools (MET), version 4.1. Given the focus on near surface conditions for renewable/demand applications, we validate against all available ground stations, which include several hundred mesonet and metar locations within the 1-km domain, covering all of Vermont and New Hampshire, along with parts of New York, Maine, and Massachusetts. Variables of interest include 10-m wind speed and direction (W), 2-m temperature (T), 2-m dew point (Td), and surface accumulated precipitation (APCP). Scores are generated based on the continuous (T, Td, W) or categorical (APCP) nature of the fields and include: mean absolute error (MAE), root mean square error (RMSE), mean error (ME), critical success index (CSI), Heidke skill score (HSS), odds ratio (OR), accuracy (ACC), probability of detection (POD), etc. This variety of metrics is chosen to create a more comprehensive, robust approach to validation.
Given the under-reporting of traditional weather stations during heavy rainfall events, among other limitations, the liquid water precipitation forecasts are validated against both the National Centers for Environmental Prediction (NCEP) gauge-corrected Stage IV and the Multi-Radar/Multi-Sensor (MRMS) gridded data, at 4.7-km and 1-km resolution, respectively. Thresholds and accumulation intervals for the contingency table analysis can then be defined according to specific applications.
To complement the quantitative verification measures, we also consider more qualitative assessment. This can include comparisons to county flood reports and visible satellite and radar imagery. Feedback is also obtained from the end users.
We will discuss the ongoing work, data, challenges, and preliminary results from the quantitative and qualitative weather model verification.