Fluvial deposits offer Earth's best-preserved geomorphic record of past climate change over geological timescales. However, quantitatively extracting this information remains challenging in part due to the complexity of erosion, sediment transport and deposition processes and how each of them responds to climate. Furthermore, sedimentary basins have the potential to temporarily store sediments, and rivers subsequently rework those sediments. This may introduce time lags into sedimentary signals and obscure any direct correlation with climate forcing. Here, using a numerical model that combines all three processes-and a new analytical solution-we show that the thickness of fluvial deposits at the outlet of a mountain river can be linked to the amplitude and period of rainfall oscillations but is modulated by the mountain uplift rate. For typical uplift rates of a few mm/yr, climate oscillations at Milankovitch periods lead to alluvial sediment thickness of tens of meters as observed in nature. We also explain the time lag of the order of 20%-25% of the forcing period that is commonly observed between the timing of maximum rainfall and erosion. By comparing to field datasets, our predictions for the thickness and time lag of fluvial deposits are broadly consistent with observations despite the simplicity of our modeling approach. These findings provide a new theoretical framework for quantitatively extracting information on past rainfall variations from fluvial deposits.