中文标题#
一种关于视频 - LLMs 如何回答视频问题的实证研究
英文标题#
An Empirical Study on How Video-LLMs Answer Video Questions
中文摘要#
利用大规模数据和预训练语言模型，视频大型语言模型（Video-LLMs）在回答视频问题方面表现出强大的能力。然而，大多数现有工作集中在提高性能上，对理解其内部机制的关注有限。本文旨在通过系统的实证研究来弥补这一差距。为了解释现有的 VideoLLMs，我们采用注意力击穿作为主要分析工具，并设计了三种变体：视频时间击穿、视频空间击穿和语言到视频击穿。然后，我们将这三种击穿应用于不同数量的层（层窗口）。通过仔细控制层窗口和击穿类型，我们提供了两种设置：全局设置和细粒度设置。我们的研究揭示了三个关键发现：(1) 全局设置表明视频信息提取主要发生在早期层，形成一个清晰的两阶段过程 —— 低层专注于感知编码，而高层处理抽象推理；(2) 在细粒度设置中，某些中间层对视频问答产生不成比例的影响，作为关键异常值，而其他大多数层贡献很小；(3) 在两种设置中，我们观察到空间 - 时间建模更多依赖于语言引导的检索，而不是视频标记之间的内部和跨帧自注意力，尽管后者计算成本较高。最后，我们证明这些见解可以用来减少 Video-LLMs 中的注意力计算。据我们所知，这是第一项系统揭示 Video-LLMs 内部如何处理和理解视频内容的工作，为未来的研究提供了可解释性和效率视角。
英文摘要#
Taking advantage of large-scale data and pretrained language models, Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions. However, most existing efforts focus on improving performance, with limited attention to understanding their internal mechanisms. This paper aims to bridge this gap through a systematic empirical study. To interpret existing VideoLLMs, we adopt attention knockouts as our primary analytical tool and design three variants: Video Temporal Knockout, Video Spatial Knockout, and Language-to-Video Knockout. Then, we apply these three knockouts on different numbers of layers (window of layers). By carefully controlling the window of layers and types of knockouts, we provide two settings: a global setting and a fine-grained setting. Our study reveals three key findings: (1) Global setting indicates Video information extraction primarily occurs in early layers, forming a clear two-stage process -- lower layers focus on perceptual encoding, while higher layers handle abstract reasoning; (2) In the fine-grained setting, certain intermediate layers exert an outsized impact on video question answering, acting as critical outliers, whereas most other layers contribute minimally; (3) In both settings, we observe that spatial-temporal modeling relies more on language-guided retrieval than on intra- and inter-frame self-attention among video tokens, despite the latter's high computational cost. Finally, we demonstrate that these insights can be leveraged to reduce attention computation in Video-LLMs. To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content, offering interpretability and efficiency perspectives for future research.
文章页面#
PDF 获取#
抖音扫码查看更多精彩内容