BayesEvolve: Uncertainty-Aware Sequential Decision-Making with LLMs

Large language models (LLMs) can generate candidates for sequential decision problems, but many LLM agents use past evaluations only as examples, without explicitly modeling the performance and uncertainty of unevaluated candidates. We introduce $\textbf{BayesEvolve}$, a closed-loop framework that converts evaluation history into a predictive belief state. BayesEvolve uses a Gaussian process to predict each candidate's objective value and uncertainty. It combines these predictions in a value--uncertainty score that prioritizes candidates for the LLM, and observed outcomes update the model. On five shifted black-box benchmarks with 100 evaluations per run, BayesEvolve achieves a lower mean normalized best-so-far objective than history-based and heuristic-memory LLM baselines and Bayesian optimization using a Gaussian process. Across 60,800 held-out predictions, the model ranks candidates reliably. Controlled ablations show that most gains come from predicted performance, while uncertainty is informative but provides only a modest additional benefit. These results support predictive beliefs as a useful interface between generative proposals and uncertainty-aware sequential decision-making.