ModelForge: Test-Guided Generation and Repair of Planning Domains from Natural Language

Research has demonstrated Large Language Models' ability to generate PDDL domains from natural language descriptions. Separate research has developed model repair algorithms that use tests, which consist of a task, a set of positive plans (sequences of actions that should be solutions), and a set of negative plans (sequences of actions that should not be solutions), to repair flawed domains. However, to date, no NL-to-PDDL pipelines can accept tests as input. In this work, we present ModelForge, a pipeline that generates PDDL domains based on natural language descriptions of a domain and tests. ModelForge combines symbolic and LLM-based repair methods to suggest targeted corrections to generated domains. We automatically evaluate domain quality using NL-based test suites, and show that ModelForge improves domain quality by 56\% and reduces token usage by 32\% compared to the current state-of-the-art baseline.