Flares are powerful energy releases occurring in stellar atmospheres. Solar flares, the most intense energy bursts in the solar system, are however hardly noticeable in the total solar luminosity. Consequently, the total amount of energy they radiate 1) remains largely unknown and 2) has been overlooked as a potential contributor to variations in the Total Solar Irradiance (TSI), i.e. the total solar flux received at Earth. Here, we report on the detection of the flare signal in the TSI even for moderate flares. We find that the total energy radiated by flares exceeds the soft X-ray emission by two orders of magnitude, with an important contribution in the visible domain. These results have implications for the physics of flares and the variability of our star.