Urban dispersion models are being evaluated using tracer measurements obtained in cities during big field campaigns (e.g., SLC Urban 2000, OKC Joint Urban 2003, NYC Madison Square Garden, NYC Midtown). One of the key input parameters for the models is the prevailing wind direction. We will show that rooftop and sodar wind direction measurements obtained in cities show significant scatter making it unclear exactly what wind direction to use as input. We will demonstrate that plume concentrations at specific points in space can change several orders of magnitude with only a 10 degree change in the prevailing wind direction. When evaluating models using paired-in-space and paired-in-time concentrations, we show that it will be very difficult to achieve the commonly-used "factor-of-two" metric. We will also discuss other sensitivities of plume transport in cities to wind direction, including how the street-level flow patterns in cities can be very robust (i.e., unchanging) as the upper-level wind direction changes, and then suddenly shift 180 degrees at critical upper-level wind directions. And how slight changes in wind direction can result in a plume being caught in the updraft on the downwind side of a tall building leading to low surface-level concentrations or being caught in the downdraft on the front side of a tall building leading to high surface-level concentrations. These sensitivities to building geometry and wind direction make it extremely challenging to do well when evaluating urban dispersion models against concentration measurements in cities.