ChemLit-QA: A human evaluated dataset for chemistry RAG tasks

The evaluation of Large-Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems, particularly in knowledge-intensive fields like chemistry, is hindered by the number of available high-quality datasets. Existing datasets are often small due to the labor-intensive nature of manual curation, or require further quality checks when generated automatically. This study addresses the need for robust scientific datasets by introducing ChemLit-QA, an open-source, expert-validated dataset with over 1,000 entries tailored for chemistry. The dataset was initially generated using an automated framework and subsequently reviewed by experts.