Interior models of giant planets traditionally assume that at a given radius (i.e., pressure) the density should be larger than or equal to the one corresponding to a homogeneous, adiabatic stratification throughout the planet (referred to as the "outer adiabat"). The observations of Jupiter's gravity field by Juno combined with the constraints on its atmospheric composition appear to be incompatible with such a profile. In this Letter, we show that the above assumption stems from an incorrect understanding of the Schwarzschild-Ledoux criterion, which is only valid on a local scale. In order to fulfill the buoyancy stability condition, the density gradient with pressure in a nonadiabatic region must indeed rise more steeply than the local adiabatic density gradient. However, the density gradient can be smaller than the one corresponding to the outer adiabat at the same pressure because of the higher temperature in an inhomogeneously stratified medium. Deep enough, the density can therefore be lower than the one corresponding to the outer adiabat. We show that this is permitted only if the slope of the local adiabat becomes shallower than the slope of the outer adiabat at the same pressure, as found in recent Jupiter models due to the increase of both specific entropy and adiabatic index with depth. We examine the dynamical stability of this structure and show that it is stable against nonadiabatic perturbations. The possibility of such an unconventional density profile in Jupiter further complicates our understanding of the internal structure and evolution of (extrasolar) giant planets.