Carbon-enhanced metal-poor (CEMP) stars are a unique resource for Galactic archaeology because they probe the properties of the First Stars, early chemical evolution, and binary interactions at very low metallicity. Comparing the fractions and properties of CEMP stars in different Galactic environments can provide us with unique insights into the formation and evolution of the Milky Way halo and its building blocks. In this work, we investigate whether directly comparing fractions of CEMP stars from different literature samples of very metal-poor ($\rm {[Fe/H]}\,\, -2.0$) stars is valid. We compiled published CEMP fractions and samples of Galactic halo stars from the past 25 years, and find that they are not all consistent with each other. Focusing on giant stars, we find significant differences between various surveys when comparing their trends of [Fe/H] versus [C/Fe] and their distributions of CEMP stars. To test the role of the analysis pipelines for low-resolution spectroscopic samples, we re-analysed giant stars from various surveys with the SSPP and FERRE pipelines. We found systematic differences in [C/Fe] of ~0.1-0.4 dex, partly independent of degeneracies with the stellar atmospheric parameters. These systematics are likely due to the different pipeline approaches, different assumptions in the employed synthetic grids, and/or the comparison of different evolutionary phases. We conclude that current biases in (the analysis of) very metal-poor samples limit the conclusions one can draw from comparing different surveys. We provide some recommendations and suggestions that will hopefully aid the community to unlock the full potential of CEMP stars for Galactic archaeology.