RULERv2: Retrieval First, Solving Next, A Bottom-Up Approach for Long-Context Evaluation

Recent progress in long-context language models has spurred development of diverse benchmarks that often test multiple skills simultaneously, making it difficult to pinpoint specific reasons for model failures. To address this, we introduce RULERv2, a benchmark with systematic difficulty progression from basic synthetic retrieval to complex problem-solving across three domains: multi-key NIAH, multi-value NIAH, and multi-doc QA.We conduct a large-scale evaluation of leading models, including seven closed-source and 24 open-weight models. Our findings reveal a notable performance gap between the two. Critically, we demonstrate that all models, including those claiming million-token context windows, exhibit performance degradation with increasing length, highlighting an unresolved challenge. Our analysis shows that explicit decomposition into a retrieve-then-solve strategy outperforms implicit, single-step approaches, and chain-of-thought reasoning enables models to discover effective decomposition autonomously. Finally, we find that even top-performing open-weight models struggle with fundamental retrieval and copying tasks, leading to degraded performance on more complex problems.