Spending the Dynamics Budget with Causally Factorized World Models

World models allocate substantial capacity to environment dynamics, even when parts of the environment evolve independently of the agent. For example, in DreamerV3, the recurrent dynamics core accounts for 53.4\% of all agent parameters, exceeding the encoder, decoder, and prediction heads combined. We ask whether this parameter burden can be reduced without degrading control performance. Exogenous transitions need not condition on actions or agent-affected state, allowing the corresponding dependencies to be removed from the recurrent dynamics. We therefore introduce a causally factorized recurrent state-space model that splits the deterministic state into an exogenous branch and an agent branch, and cuts the action and the agent branch out of the exogenous mechanism. This factorization is a drop-in recurrent core with 17.6\% fewer dynamics parameters at the original state width. To go further than reporting the saving, we pair a theory of when such a cut is admissible with an ablation matrix designed against it. Experiments on Atari100k and Procgen show that the factorized model gains over the monolithic baseline with fewer dynamics parameters. Moreover, the theory is borne out both by the ablation matrix and by a separate test of the exogeneity assumption itself. These results indicate that exogenous transition structure can substitute for part of the recurrent model capacity, enabling smaller world models design.