Q-JEPA: Joint-Embedding Predictive Adaptation for Quantized Large Language Models

Low-rank adaptation and low-bit quantization make large language model fine- tuning feasible on a single consumer GPU, the dominant compute regime for most researchers worldwide. LLM-JEPA improves next-token prediction with a joint- embedding predictive objective computed over paired hidden representations, yet quantization alters the very weights that produce those representations, leaving it unclear whether such an objective retains its value once the pretrained model is compressed. We address the question with Q-JEPA, a controlled study crossing next-token and JEPA-augmented training with BF16 LoRA and 4-bit NF4 QLoRA under matched optimization. On Llama-3.2-1B-Instruct and NL-RX-SYNTH, joint-embedding supervision improves exact match for all five matched fine-tuning seeds under both precisions. We argue that this robustness plausibly follows from the form of the objective, which constrains direction in embedding space while quantization error falls mainly on magnitude, and we contribute a reusable protocol for testing whether an objective survives compression.