Post-training Conditionals: Parametric Estimator-risk for Protein Language Models

Generative protein models are often trained on data, aggregated over heterogeneous conditions. Yet, experiment results are not intrinsic properties of the protein sequence alone, but instead are conditional on the assay and the application environment. Post-training reduces this discrepancy, though its evaluation commonly relies on context-agnostic empirical losses. In this work, we add condition-awareness to protein language models accounting for the interplay of subsequent training stages leveraging an analytic Gibbs posterior over the history of subsequent training losses. While post-training stages can have different objectives, their ensemble can, under assumptions, be viewed as one statistical object allowing us to assess calibration across the recorded post-training trajectories. We empirically assess the conditioning of protein language models, their post-training stages, and model certainty as distinct sources of error in generative-model learning.