For decades, numerous diagnostic variables have been used to assess severe weather environments and to discriminate severe weather phenomena (e.g., convective mode, observed type of severe weather, tornadic versus nontornadic supercells, etc.). Many of these diagnostic variables have been introduced without sufficient validation, in terms of their capability to discriminate certain phenomena, as statistically significant improvements upon predecessor methods, or even as useful alternative options for assessments of severe weather environments. This study discusses some of the limitations commonly observed when validating diagnostic variables, using a sample of 4057 severe weather outbreaks from 1979-2008. These challenges include but are not limited to substantial correlations with other well-known diagnostic variables, improper statistical significance testing, small samples available for analysis, and incomplete skill assessment. These challenges are observed and frequently neglected in most severe weather discrimination studies, as selected examples illustrate.