Adaptive Financial Time Series Tokenization with Bilevel Optimization

Financial time series are nonstationary and heteroskedastic, making fixed interval tokenization a poor fit because quiet and volatile periods can produce tokens with very different information content. Event time sampling addresses this problem by creating a new token when accumulated market activity crosses a threshold. However, that threshold is usually fixed in advance and must balance two competing goals: uniform volatility content and regular token duration. We formulate this tradeoff as a bilevel optimization problem. A small neural boundary predictor learns when to cut the series, with the leader parameters targeting uniform volatility across tokens and a scalar follower parameter targeting uniform duration. We motivate the equal risk objective theoretically. Using the Dambis Dubins Schwarz representation, we show why sampling at approximately equal increments of quadratic variation produces approximately i.i.d. Gaussian martingale innovations and reduces the variance of realized volatility and related tail risk estimates. The effectiveness of the bilevel formulation depends critically on how the follower and its hypergradient are handled. Truncated unrolling can weaken the indirect gradient, while an overparameterized follower requires regularization that suppresses the same coupling. Restricting the follower to a single cut bias lets us solve for the hypergradient without truncation and preserve a meaningful bilevel effect. Experiments on regime switching simulations and held out equity data show that this formulation produces token boundaries that differ systematically from those of a matched single level baseline and achieves a more balanced tradeoff between volatility and duration. These results suggest that, for learned time series tokenization, the design of the follower is as important as the choice of hypergradient method.