Put Your Skills To Work: Chaining Unsupervised Skills as a Random Walk for Exploration

Self-supervised pre-training in reinforcement learning aims to discover a diverse set of behaviours without extrinsic reward, which can then accelerate a wide range of downstream tasks. Recent skill discovery methods such as METRA and CSF learn skills as \emph{directions} in a latent space and produce impressive behaviours in locomotion domains. However, the exploration and hence the state coverage of existing skill learning methods degrades sharply in more complex environments, and existing remedies add machinery on top of skill learning, such as an explicit skill tree or a learned guide policy. Leveraging the latent space METRA/CSF already learn, we instead change only how skills are executed during training. Rather than committing to one skill per episode, we chain skills of a fixed horizon, resampling the latent direction at each boundary, so that a rollout performs a random walk whose steps are skill executions rather than primitive actions. Applying this to CSF gives Chained-CSF (CCSF), which leaves the base algorithm, its losses and its networks untouched and adds only the skill horizon as a hyperparameter. On a bottleneck maze, chaining raises state coverage from $51.6\%$ to $94.7\%$ over unmodified CSF at the same environment-step budget, exceeding a method designed specifically to overcome such bottlenecks. On a modified Craftax-Classic environment in which survival pressures are removed, CCSF climbs the crafting tree to iron in $97.9\%$ of episodes and obtains an iron pickaxe and diamonds \textbf{without any task rewards}, while unmodified CSF does not reliably collect stone. The skills learned during pre-training are also directly usable: without any reward or additional learning, goals specified only as a desired change in the raw observation can be converted into latent directions and commanded from the frozen policy, reaching every stage up to iron in over $90\%$ of episodes and doing so several times faster than random skill selection.