SWIM can provide global wave spectra, but under small sea conditions, the presence of parasitic peaks at low wavenumber, surfboard effects, and residual speckle noise lead to performance degradation of SWIM wave height spectrum products. To reduce the impacts of the factors above on SWIM wave height spectrum, in this paper, a convolution neural network (CNN) method based on BU-Net is proposed for calibrating SWIM omnidirectional wave height spectra with buoy measurements under sea states (wind wave mainly/swell mainly) and sea surface conditions (wind speed from 9m/s to 19m/s, significant wave height from 0.8m to 3.4m∼4.2m). The calibration results show that the impact of the factors above on SWIM omnidirectional wave height spectrum can be corrected. The correlation coefficients between the corrected SWIM beams 6°, 8°, 10 °and the buoy mean omnidirectional wave height spectrum are all greater than 0.90, and the relative error of the peak wave number is within 10%. The relative error of the integrated energy is mostly less than 20%. In addition, the performance of spectral integration parameters (effective wave height Hs, energy wave period Tm-10) of each spectral beam of SWIM has been verified using MFWAM reanalysis data. The validation results show that RMSE of the Hs and Tm-10, for the corrected SWIM beam 6 °(8 °, 10 °) under wind wave sea conditions are 0.31m (0.32m, 0.25m) and 0.50s (0.51s, 0.49s) respectively; those for swell cases are 0.16m (0.16m, 0.13m) and 0.87s (0.77s, 0.72s), respectively.