In spite of decades of theoretical efforts, the physical origin of the stellar initial mass function (IMF) is still debated. Particularly crucial is the question of what sets the peak of the distribution. To investigate this issue, we perform high-resolution numerical simulations with radiative feedback exploring, in particular, the role of the stellar and accretion luminosities. We also perform simulations with a simple effective equation of state (EOS), and we investigate 1000 solar-mass clumps having, respectively, 0.1 and 0.4 pc of initial radii. We found that most runs, both with radiative transfer or an EOS, present similar mass spectra with a peak broadly located around 0.3-0.5 M_⊙ and a power-law-like mass distribution at higher masses. However, when accretion luminosity is accounted for, the resulting mass spectrum of the most compact clump tends to be moderately top-heavy. The effect remains limited for the less compact one, which overall remains colder. Our results support the idea that rather than the radiative stellar feedback, this is the transition from the isothermal to the adiabatic regime, which occurs at a gas density of about 10¹⁰ cm^-3, that is responsible for setting the peak of the IMF. This stems from (i) the fact that extremely compact clumps for which the accretion luminosity has a significant influence are very rare and (ii) the luminosity problem, which indicates that the effective accretion luminosity is likely weaker than expected.