Combining datasets with different ground truths using Low-Rank Adaptation to generalize image-based CNN models for photometric redshift prediction

In this work, we demonstrate how Low-Rank Adaptation (LoRA) can be used to combine different galaxy imaging datasets to improve redshift estimation for cosmology with CNN models. LoRA is an established technique for large language models that adds adapter networks to adjust model weights and biases to efficiently fine-tune large base models without retraining. We train a base model using a photometric redshift ground truth dataset, which contains broad galaxy types but is less accurate. We then fine-tune using LoRA on a spectroscopic redshift ground truth dataset. These are more accurate but limited to bright galaxies and take orders of magnitude more time, so are less available for large datasets. The combination of the two datasets is ideal for training accurate models that can generalize well. The LoRA model performs better than a traditional transfer learning method, with $\sim$6.5$\times$ less bias and $\sim$2.4$\times$ less scatter. Retraining the model on a combined dataset performs better than LoRA but at a cost of greater computation time. Our work shows that LoRA is useful for fine-tuning regression models in astrophysics, and allows us to leverage existing pretrained models for future large cosmological sky surveys, which are too large for complete spectroscopic coverage.