Beyond Coarse Blocks: Fine-Grained Sparse Attention for Video Diffusion

Video Sparse Attention (VSA) accelerates video diffusion by applying exact attention to selected regular blocks, but its 64-token support granularity can discard locally important interactions. We introduce a training-free refinement that scores 8-token sub-blocks while preserving the selected valid-token count of the released 80%-sparse VSA configuration for every query block. Across a matched, fixed-seed evaluation of all 946 VBench prompts and 16 dimensions, fine-grained support improves normalized Total by +1.38 points to 81.59 and improves 11 of 16 dimensions at the same sparsity. In an isolated same-prompt run, the current packed executor adds +4.27% end-to-end latency relative to coarse VSA. Profiling localizes its 2.48× attention-path gap to the unfused execution of short, irregular segments, leaving custom fusion and scheduling as a well-scoped next systems step.