For the case of Uranus and Neptune we show that current knowledge of the fourth zonal gravity harmonic, J4, constrains observed zonal jets to a thin weather-layer containing no more than the uppermost 0.2% of the planetary mass. This means that the jets have an e-folding decay depth of no more than 1000 km. For Jupiter and Saturn, more accurate gravity data is required to calculate precise limits. This is likely to be obtained in 2016 when the Juno and Cassini spacecraft perform close flybys of these planets, measuring for the first time the high order gravity spectrum of these planets. While the low order gravity harmonics are dominated by the oblateness and radial mass distribution of the planets, the high (n>10) and odd gravity harmonics contain information about the dynamics, via the dynamical balance between the density anomalies and the flow field, and north-south hemispherical asymmetries in the circulation. We will discuss how these constraints on the interior flow affect the possible driving mechanisms of the atmospheric jets.