A Query-Conditioned Geometry for Long-Form Video Understanding

Multimodal LLMs (MLLMs) cannot process a long video in full. Minutes to hours of a long-form sequence exceed any context window and a larger window size does not solve the problem, because irrelevant frames dilute attention away from the few that carry the relevant information. Model receive their context from a preselected subset of frames or patches from the video. When the information and scene that answer a question fall outside that subset, the model loses that context and will answer from whatever is present in its context and learned knowledge. Given a long video and a question $q$, choosing which $K$ frames enter that context is the problem we study.