Do Linguistic Features Help Code-Switching Preference Models?

LLMs may already know a great deal about bilingual grammar, but a learned reward model may fail to use—or preserve—that knowledge during optimization. This matters for code-switching: speakers follow grammatical patterns but vary in where and how they switch, while native-speaker comparisons of alternative Hinglish sentences are costly to collect. Simple mixing measures correlate only weakly with acceptability, and LLM scores often disagree with native speakers on Hindi-English verb choices. We therefore ask a practical question: when labels are scarce, do theory-motivated linguistic features help a reward model generalize to held-out examples from the same domain? We test this with a shuffled-feature control and a probe for generator-specific clues. Future work will ask whether the same features reduce shortcut learning during reinforcement-learning optimization. Data and models: HinGE contains 1,976 Hindi-English sources, each with a word-aligned (WAC) and a phrase-aligned (PAC) code-mixed candidate. We introduce Ling, a model that adds theory-motivated linguistic metrics derived from HingBERT language tags. These metrics describe mixing amount, switch and span patterns, source compatibility, and tag confidence. Text uses only XLM-R representations. Ling adds the matched linguistic metrics, while Shuffled adds the same metrics but randomly assigns them to candidates within each split. This control tests whether the correct linguistic information helps rather than simply giving the model more inputs. Positive differences favor Ling. Evaluation and statistics: We average the two human ratings for each candidate. For every source, the model ranks its WAC and PAC versions. Pair accuracy is the fraction ranked in the same order as the human ratings. We train on 100, 250, or 500 candidates, corresponding to 50, 125, or 250 source pairs. At each size, we use five different training samples. We run five-fold cross-validation twice, keeping both candidates and any duplicate sources in the same split. A separate development set chooses the regularization setting, and the test set is used only for final scoring. We estimate 95% confidence intervals with 2,000 bootstrap samples. Area under the learning curve on a log training scale summarizes performance across the three training sizes, while Holm correction covers four feature-family comparisons. For the generator probe, an ROC-AUC of 0.5 represents chance performance. Results: Ling has the highest mean pair accuracy at all three training sizes. At $N=100$, Text, Shuffled, and Ling obtain accuracies of 0.556, 0.554, and 0.568. At $N=250$, they obtain 0.565, 0.571, and 0.586. At $N=500$, they obtain 0.586, 0.584, and 0.597. At $N=250$, Ling improves over Text by 0.021 with a 95% confidence interval of [0.005, 0.037]. This is the only individual training size where the Ling–Text interval excludes zero, and all individual Ling–Shuffled intervals include zero. Across training sizes, Ling improves log-scale area under the learning curve by 0.016, with a 95% confidence interval of [0.006, 0.028], over Text and by 0.014, with an interval of [0.001, 0.027], over Shuffled. The largest feature-family gain comes from language-identification uncertainty, although its Holm-adjusted $p$-value of 0.052 is just above 0.05. The Ling features also predict generator identity with an ROC-AUC of 0.737. Thus, on HinGE, the features carry a modest preference signal alongside generator-specific clues. Future work: We will use linguistic reward metrics to create preference pairs for code-switching generation. We will add richer grammatical, part-of-speech, and span-based features, then compare text-only, linguistic-only, and hybrid rewards across label budgets on HinGE and an independent verb-choice probe. We will test generalization on Telugu-English medical dialogues and other code-mixed datasets. From the same supervised model, we will compare group-relative reinforcement learning and preference-based reward shaping, tracking human agreement, meaning, diversity, feature drift, and reward exploitation. We expect hybrid rewards to help most when few labels are available.