Do LLM Agents Behave Like Different People? Human-Calibrated Behavioural Measurement with LLM-Scored Psychometrics

Giving LLM agents different personas does not guarantee that they behave like different people. Testing persona fidelity requires measuring how much individuals actually differ in their behaviour, while accounting for the situations that elicit that behaviour. We present LIRT (LLM-scored item response theory), a framework for constructing such measurements from question--answer transcripts. LIRT uses a separate LLM-based coder to score each exchange against a fixed battery of binary behavioural items, then maps those codes onto shared behavioural dimensions using a contextual item response model, a traditional psychometric tool. Crucially, the model adjusts the differences between responders for the differences in the situations they face, providing a consistent axis on which to compare them. We calibrate LIRT on 15,252 analyst--CEO exchanges from 100 public-company earnings calls and recover five interpretable behavioural axes. The situation accounts for substantially more variation in observed behaviour than the individual, but individual differences remain stable, with measurements of the same person from split-half exchanges reproducing at $r=0.90$. This gives a human reference for quantifiable behaviour, where LLM agents can be scored on the same five axes and compared with the variation observed across real people. We also test how reliably those axes can be constructed. Agreement between two LLM coder families is moderate (Cohen's $\kappa=0.62$ and $0.55$) and falls on items requiring a more nuanced understanding of behaviour, for example, sarcasm. Any human--agent comparison therefore inherits uncertainty from the coder itself. Applied to agent populations, LIRT could test whether persona prompts produce stable behavioural differences, and how large those differences are relative to human variation.