The Time is Here for Just-in-Time Systems: Challenges and Opportunities

LLM coding agents can now write, debug, and refactor systems of tens of thousands of lines, opening a new form of design-space exploration: synthesizing an entire system from scratch, specialized to a given environment, workload, and set of required properties. We call these Just-in-Time (JIT) systems and present Jitskit, an agentic pipeline that turns three spec cards into a complete key-value store through a design--evaluate--reflect loop with three additions to a bare coding agent: an adversarial auditor that hunts for reward hacks and grows the test suite, an instrumented evaluator that exposes leading indicators rather than a single throughput number, and a whiteboard memory of ruled-out designs. Across 18 specs, Jitskit's generated key-value stores outperform the best of FASTER, F2, RocksDB, and Redis by up to 4.6x, choosing a different architecture per regime. Claude Code alone reward-hacks or trails by up to 5.4x, and removing leading-indicator feedback costs up to 3.75x. We distill three lessons for agentic systems design: specifications are never complete, many performance wins are correctness hacks in disguise, and the evaluator is the loss function.