The detection of exoplanets in coronographic images is severely limited by residual starlight speckles. Dedicated post-processing can drastically reduce this 'stellar leakage' and thereby increase the faintness of detectable exoplanets. Based on a multispectral series expansion of the diffraction pattern, we derive a multimode model of the residuals which can be exploited to estimate and thus remove the residual speckles in multispectral coronographic images. Compared to other multispectral processing methods, our model is physically grounded and is suitable for use in an (optimal) inverse approach. We demonstrate the ability of our model to correctly estimate the speckles in simulated data and demonstrate that very high contrasts can be achieved. We further apply our method to removing speckle from a real data cube obtained with the SPHERE integral field spectrograph instrument.