We present the first measurements of the Lyman-continuum photon production efficiency ξ_{ion, 0} at z ∼ 4-5 for galaxies fainter than 0.2 L^* (-19 mag). ξ_{ion, 0} quantifies the production rate of ionizing photons with respect to the UV luminosity density assuming a fiducial escape fraction of zero. Extending previous measurements of ξ_{ion, 0} to the faint population is important, as ultra-faint galaxies are expected to contribute the bulk of the ionizing emissivity. We probe ξ_{ion, 0} to such faint magnitudes by taking advantage of 200-h depth Spitzer/IRAC observations from the GREATS program and ≈300 3 < z < 6 galaxies with spectroscopic redshifts from the MUSE GTO Deep + Wide programs. Stacked IRAC [3.6]-[4.5] colors are derived and used to infer the Hα rest-frame equivalent widths, which range from 403 Å to 2818 Å. The derived ξ_{ion, 0} is log₁₀(ξ_ion,0/Hz erg^-1) = 25.36 ± 0.08 over -20.5 < M_UV < -17.5, similar to those derived for brighter galaxy samples at the same redshift and therefore suggesting that ξ_ion shows no strong dependence on M_UV. The ξ_{ion, 0} values found in our sample imply that the Lyman-continuum escape fraction for M_UV ≈ -19 star-forming galaxies cannot exceed ≈8-20% in the reionization era.