中文标题#
一種關於視頻 - LLMs 如何回答視頻問題的實證研究
英文标题#
An Empirical Study on How Video-LLMs Answer Video Questions
中文摘要#
利用大規模數據和預訓練語言模型，視頻大型語言模型（Video-LLMs）在回答視頻問題方面表現出強大的能力。然而，大多數現有工作集中在提高性能上，對理解其內部機制的關注有限。本文旨在通過系統的實證研究來彌補這一差距。為了解釋現有的 VideoLLMs，我們採用注意力擊穿作為主要分析工具，並設計了三種變體：視頻時間擊穿、視頻空間擊穿和語言到視頻擊穿。然後，我們將這三種擊穿應用於不同數量的層（層窗口）。通過仔細控制層窗口和擊穿類型，我們提供了兩種設置：全局設置和細粒度設置。我們的研究揭示了三個關鍵發現：(1) 全局設置表明視頻信息提取主要發生在早期層，形成一個清晰的兩階段過程 —— 低層專注於感知編碼，而高層處理抽象推理；(2) 在細粒度設置中，某些中間層對視頻問答產生不成比例的影響，作為關鍵異常值，而其他大多數層貢獻很小；(3) 在兩種設置中，我們觀察到空間 - 時間建模更多依賴於語言引導的檢索，而不是視頻標記之間的內部和跨幀自注意力，儘管後者計算成本較高。最後，我們證明這些見解可以用來減少 Video-LLMs 中的注意力計算。據我們所知，這是第一項系統揭示 Video-LLMs 內部如何處理和理解視頻內容的工作，為未來的研究提供了可解釋性和效率視角。
英文摘要#
Taking advantage of large-scale data and pretrained language models, Video Large Language Models (Video-LLMs) have shown strong capabilities in answering video questions. However, most existing efforts focus on improving performance, with limited attention to understanding their internal mechanisms. This paper aims to bridge this gap through a systematic empirical study. To interpret existing VideoLLMs, we adopt attention knockouts as our primary analytical tool and design three variants: Video Temporal Knockout, Video Spatial Knockout, and Language-to-Video Knockout. Then, we apply these three knockouts on different numbers of layers (window of layers). By carefully controlling the window of layers and types of knockouts, we provide two settings: a global setting and a fine-grained setting. Our study reveals three key findings: (1) Global setting indicates Video information extraction primarily occurs in early layers, forming a clear two-stage process -- lower layers focus on perceptual encoding, while higher layers handle abstract reasoning; (2) In the fine-grained setting, certain intermediate layers exert an outsized impact on video question answering, acting as critical outliers, whereas most other layers contribute minimally; (3) In both settings, we observe that spatial-temporal modeling relies more on language-guided retrieval than on intra- and inter-frame self-attention among video tokens, despite the latter's high computational cost. Finally, we demonstrate that these insights can be leveraged to reduce attention computation in Video-LLMs. To our knowledge, this is the first work to systematically uncover how Video-LLMs internally process and understand video content, offering interpretability and efficiency perspectives for future research.
文章页面#
PDF 获取#
抖音掃碼查看更多精彩內容