SafeSciBench: An Artifact-Controlled Safety Benchmark for AI4Science Agents

Scientific AI agents now plan experiments, run code, retrieve literature, and interpret results, with a human reviewing only a fraction of what they do. Their safety in this open loop rests on the knowledge that gates their actions, yet the safety evaluations applied to them were built for general-purpose chatbots and for catastrophic-capability testing. Those instruments miss the failures that occur when a scientific agent acts on flawed knowledge of misuse, integrity, or dual-use. We present SafeSciBench (SSB), an 840-question multiple-choice benchmark for AI4Science safety, organized across four risk axes and seven risk categories and stratified by Bloom's cognitive levels. General multiple-choice benchmarks leak the answer through two surface cues, and the unprocessed source set carried both: a position cue, where one option letter held about 96\% of correct answers, and a verbosity cue, where the correct option was the longest in about 99\% of items. SSB removes both. A seeded balance-shuffle fixes each letter at exactly 25\% of correct answers, and a freeze-correct rewriting pass holds the correct text fixed while rewriting distractors, so the correct option is the longest in 0.6\% of items. We evaluate three open-weight models spanning 8B to 120B parameters (Llama-3.1-8B, GPT-OSS-120B, Qwen3-32B) on SSB and on WMDP, a standard hazardous-knowledge benchmark, through the UK AISI Inspect framework. The two instruments rank the top models in different order. WMDP places GPT-OSS-120B first and SSB places Qwen3-32B first, though this top-two reordering is not statistically robust, and both instruments agree that Llama-3.1-8B is weakest. The divergence is sharper by category: the weakest SSB category, Negative Dual-Use, holds 39.5\% of the items no model solves. The item keys come from construction metadata and an automated integrity pipeline, not an independent human audit. We report confidence intervals, per-position diagnostics, and item-difficulty statistics. SSB gives a team a portable instrument for vetting the safety-relevant knowledge of a scientific agent before it runs in an open loop, on an API budget and without sandbox infrastructure.