We use spectral energy distribution fitting to place constraints on the stellar populations of 59 ultra-diffuse galaxies (UDGs) in the low-to-moderate density fields of the MATLAS survey. We use the routine PROSPECTOR, coupled with archival data in the optical from the Dark Energy Camera Legacy Survey, and near- and mid-infrared imaging from the Wide-field Infrared Survey Explorer, to recover the stellar masses, ages, metallicities, and star formation time-scales of the UDGs. We find that a subsample of the UDGs lies within the scatter of the mass-metallicity relation (MZR) for local classical dwarfs. However, another subsample is more metal-poor, being consistent with the evolving MZR at high redshift. We investigate UDG positioning trends in the mass-metallicity plane as a function of surface brightness, effective radius, axis ratio, local volume density, mass-weighted age, star formation time-scale, globular cluster (GC) counts, and GC specific frequency. We find that our sample of UDGs can be separated into two main classes: Class A: comprised of UDGs with lower stellar masses, prolonged star formation histories (SFHs), more elongated, inhabiting less dense environments, hosting fewer GCs, younger, consistent with the classical dwarf MZR, and fainter. Class B: UDGs with higher stellar masses, rapid SFHs, rounder, inhabiting the densest of our probed environments, hosting on average the most numerous GC systems, older, consistent with the high-redshift MZR (i.e. consistent with early-quenching), and brighter. The combination of these properties suggests that UDGs of Class A are consistent with a 'puffed-up dwarf' formation scenario, while UDGs of Class B seem to be better explained by 'failed galaxy' scenarios.