JSLBench: Evaluating LLM Performance for a Rare Coding Language in JMP

Large language models (LLMs) are becoming increasingly good at generating data-analysis code, yet their reliability in domain-specific languages remains poorly characterized. As a powerful statistical software, JMP supports a broad range of analyses, from general tasks like data tables manipulation, modeling, visualization, dashboards construction, to specialized platforms like design of experiments (DOE), and quality engineering. These capabilities are used by scientists and engineers in pharmaceutical, chemical, semiconductor, and manufacturing R&D, where reproducible analysis, process optimization, and quality monitoring are central. Its scripting language, JMP Scripting Language (JSL), allows users to automate these analyses and package them as reusable scripts and add-ins for deployment across teams. Despite its popular role in industry, publicly accessible JSL code is comparatively limited relative to widely used statistical languages such as R and SAS. More importantly, no established benchmark evaluates whether LLMs can generate reliable JSL across JMP’s analytical ecosystem. We present JSLBench, the first benchmark for natural-language-to-JSL generation. JSLBench contains 10K tasks drawn from sample analyses, community forums, and testing workflows. Each task pairs a natural-language request with a reference JSL script and a sample dataset distributed with JMP. All reference scripts were executed and verified in JMP 19.1. We create JSLBench data in a three-step process. First, two frontier LLMs (GPT 5.5 and Claude Opus 4.8) generate a natural-language request from verified scripts without exposing JSL function or platform names. Then we do quality assurance to detect exact and semantic duplicates and decomposes multi-goal requests into focused single-goal prompts, with 45 duplicate tasks removed and 58 multi-goal requests split. Finally, we evaluate prompt–script consistency, analytical coverage, clarity, and concision with LLM revision. To reduce cost while maintaining broad task coverage, we construct JSLBench-Diverse, a 500-task subset selected through diversity sampling in TF-IDF space to broaden coverage of JMP platforms and functions. We evaluate popular LLMs on JSLBench-Diverse. All runs follow a single-attempt evaluation protocol, which defines clean execution as completion in JMP without errors, warnings, modal dialogs, or timeouts. Codex with GPT-5.5 at high reasoning effort achieves an 83% clean-execution rate, followed by Claude Code with Opus 4.8 at high reasoning effort (75%) and Claude Code with Sonnet 4.6 at low reasoning effort (53%). The wide range of clean-execution rates revealing substantial differences across models. Error analysis identifies recurrent failures including hallucinated JSL functions, invalid analysis specifications. Furthermore, we demonstrate how JSLBench can help in developing task-specific agents. To do this, we present its application on DOE agent development. Using the same base model and 100 DOE tasks, three candidate prompts ranges from 85% to 90% clean execution rate, improving upon the 83% general-agent baseline. JSLBench and its evaluation pipeline provide a foundation for studying reliable code generation and statistical agents in underrepresented analytical languages.