Separating Solvers and Evaluators Is Not Enough: Peer-Reward Misalignment in Multi-Agent LLMs

Language model systems increasingly use other models to evaluate, supervise, and generate learning signals. A common safeguard is to assign the solver and evaluator roles to different models. We ask whether this is sufficient when the evaluator’s judgment still determines the solver’s reinforcement-learning reward. We train two independently optimized language-model agents that solve separate tasks and grade one another, and compare a reward-coupled condition, in which each agent learns from the grade assigned by its peer, against a matched reward-decoupled condition, in which the same peer grading occurs but learning uses an external task metric. Across three 7–9B models, six tasks, and two policy-optimization algorithms, giving 36 matched pairs, peer-assigned scores exceed the external task metric in 25 of 36 reward-coupled runs, and reward decoupling improves externally measured task performance in 33 of 36 pairs. Peer-assigned grades are higher under coupling in only 19 of 36, so the coupled–decoupled difference is driven more by degraded task performance than by more generous grading. Divergence between peer scores and the external metric is concentrated on summarization. We interpret this as peer-reward-induced misalignment: solver–evaluator role separation is a weaker safeguard than reward-channel separation. We do not establish collusion, and our design does not isolate reciprocity as the causal mechanism.