We present simple stellar population models based on the empirical X-shooter Spectral Library (XSL) from near-ultraviolet (NUV) to near-infrared (NIR) wavelengths. The unmatched characteristics of the relatively high resolution and extended wavelength coverage (350-2480 nm, R ∼ 10 000) of the XSL population models bring us closer to bridging optical and NIR studies of intermediate-age and old stellar populations. It is now common to find good agreement between observed and predicted NUV and optical properties of stellar clusters due to our good understanding of the main-sequence and early giant phases of stars. However, NIR spectra of intermediate-age and old stellar populations are sensitive to cool K and M giants. The asymptotic giant branch, especially the thermally pulsing asymptotic giant branch, shapes the NIR spectra of 0.5-2 Gyr old stellar populations; the tip of the red giant branch defines the NIR spectra of older populations. We therefore construct sequences of the average spectra of static giants, variable O-rich giants, and C-rich giants to be included in the models separately. The models span the metallicity range −2.2 < [Fe/H] < +0.2 and ages above 50 Myr, a broader range in the NIR than in other models based on empirical spectral libraries. We focus on the behaviour of colours and absorption-line indices as a function of age and metallicity. Our models can reproduce the integrated optical colours of the Coma cluster galaxies at the same level as other semi-empirical models found in the literature. In the NIR, there are notable differences between the colours of the models and Coma cluster galaxies. Furthermore, the XSL models expand the range of predicted values of NIR indices compared to other models based on empirical libraries. Our models make it possible to perform in-depth studies of colours and spectral features consistently throughout the optical and the NIR range to clarify the role of evolved cool stars in stellar populations.