Observations suggest that star formation in filamentary molecular clouds occurs in a two-step process, with the formation of filaments preceding that of prestellar cores and stars. Here, we apply the gravoturbulent fragmentation theory of Hennebelle & Chabrier to a filamentary environment, taking into account magnetic support. We discuss the induced geometrical effect on the cores, with a transition from 3D geometry at small scales to 1D at large ones. The model predicts the fragmentation behavior of a filament for a given mass per unit length (MpL) and level of magnetization. This core mass function (CMF) for individual filaments is then convolved with the distribution of filaments to obtain the final system CMF. The model yields two major results. (I) The filamentary geometry naturally induces a hierarchical fragmentation process, first into groups of cores, separated by a length equal to a few filament Jeans lengths, I.e., a few times the filament width. These groups then fragment into individual cores. (II) Non-magnetized filaments with high MpL are found to fragment excessively, at odds with observations. This is resolved by taking into account the magnetic field (treated simply as additional pressure support). The present theory suggests two complementary modes of star formation: although small (spherical or filamentary) structures will collapse directly into prestellar cores, according to the standard Hennebelle-Chabrier theory, the large (filamentary) ones, the dominant population according to observations, will follow the aforedescribed two-step process.