Model-Free Assessment of Simulator Fidelity via Quantile Curves

Simulation is now pervasive, arising from manufacturing to LLM-driven applications in research, education, and consumer surveys. Yet, fully characterizing the discrepancy between simulators and ground truth remains challenging. We propose a computationally tractable method to estimate the quantile function of the discrepancy between the simulated and ground-truth distributions. The approach does not impose any modeling assumptions on the simulator and it applies broadly across many parameter families: from Bernoulli and multinomial to continuous, vector-valued settings. The resulting quantile curve supports risk-aware summaries (e.g., VaR/CVaR) and comparison of simulators or prompts performance. We illustrate our framework through an application assessing LLM simulation fidelity on the OpinionQA dataset, augmented with simulations spanning seven LLMs.