STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

While alleviating the quadratic complexity of Softmax attention is crucial, training large linear models from scratch is computationally prohibitive. Thus, linearizing pretrained LLMs via intra-layer hybrid architectures has emerged as the indispensable paradigm. Existing methods perform token routing based on sliding-window partitions, resulting in position-based selection and fails to capture token-specific global importance. Meanwhile, linear attention further suffers from distribution shift caused by learnable feature maps that distort pretrained feature magnitudes. Motivated by these limitations, we propose STILL, an intra-layer hybrid linearization framework for efficiently linearizing LLMs. STILL introduces a Self-Saliency Score with strong local–global consistency, enabling accurate token selection using sliding-window computation, and retains salient tokens for sparse softmax attention while summarizing the remaining context via linear attention. To preserve pretrained representations, we design a Norm-Preserved Feature Map (NP-Map) that decouples feature direction from magnitude and reinjects pretrained norms. We further adopt a unified training–inference architecture with chunk-wise parallelization and delayed selection to improve hardware efficiency. Experiments show that STILL matches or surpasses the original pretrained model on commonsense and general reasoning tasks, and achieves up to a 86.2\% relative improvement over prior linearized attention methods on long-context benchmarks. Source code can be found in the supplementary materials.