Dimensions of Generative AI Evaluation Design

Evaluating the capabilities and risks of generative AI (GenAI) models and systems is crucial for their successful development, deployment, and adoption, but there are few well-understood principles or guidelines for ensuring effective evaluations. To address this, we propose a set of general dimensions that capture critical choices involved in GenAI evaluation design. These dimensions include the evaluation setting, object of the evaluation, task type, input source, interaction style, duration, metric type, and scoring method. By situating evaluations within these dimensions, we aim to guide decision-making during GenAI evaluation design and provide a structure for comparing different evaluations. We illustrate the utility of these dimensions through examples, including evaluations of fairness and biological threats. Our proposal encourages a more methodical and explicit approach to GenAI evaluation design.