MedBrowseComp: Benchmarking Medical Deep Research and Computer Use

Large language models (LLMs) are increasingly envisioned as decision-support tools in clinical practice, yet safe clinical reasoning demands the integration of heterogeneous knowledge bases—trials, primary studies, regulatory documents, and cost data—under strict accuracy constraints. Existing evaluations typically rely on synthetic prompts, reduce the task to single-hop factoid queries, or conflate reasoning with open-ended text generation, leaving their real-world utility unclear. To close this gap, we present \textbf{MedBrowseComp}, the first benchmark that systematically tests an agent’s ability to reliably retrieve and synthesize multi-hop medical facts from up-to-date, domain-specific knowledge bases. MedBrowseComp holds 1,000+ human-curated questions that mirror clinical scenarios in which practitioners must reconcile information fragmented over many sources that are potentially conflicting. Applying MedBrowseComp to frontier agentic systems reveals \textbf{marked performance shortfalls as low as 10\%}. MedBrowseComp reveals critical gaps between current LLM performance and clinical usage, providing a testbed to guide future model and toolchain improvements for reliable medical information seeking.