Stellar magnetic activity induces both distortions and Doppler-shifts in the absorption line profiles of Sun-like stars. Those effects produce apparent radial velocity (RV) signals which greatly hamper the search for potentially habitable, Earth-like planets. In this work, we investigate these distortions in the Sun using cross-correlation functions (CCFs), derived from intensive monitoring with the high-precision spectrograph HARPS-N. We show that the RV signal arising from line-shape variations on time-scales associated with the Sun's rotation and activity cycle can be robustly extracted from the data, reducing the RV dispersion by half. Once these have been corrected, activity-induced Doppler-shifts remain, that are modulated at the solar rotation period, and that are most effectively modelled in the time domain, using Gaussian processes (GPs). Planet signatures are still best retrieved with multidimensonal GPs, when activity is jointly modelled from the raw RVs and indicators of the line width or of the Ca II H & K emission. After GP modelling, the residual RVs exhibit a dispersion of 0.6-0.8 m s^-1, likely to be dominated by signals induced by supergranulation. Finally, we find that the statistical properties of the RVs evolve significantly over time, and that this evolution is primarily driven by sunspots, which control the smoothness of the signal. Such evolution, which reduces the sensitivity to long-period planet signatures, is no longer seen in the activity-induced Doppler-shifts, which is promising for long term RV monitoring surveys such as the Terra Hunting Experiment or the PLATO follow-up campaign.