Abstract
We introduce Hierarchical Streaming Video Understanding, a task that combines online temporal action localization with free-form description generation. Given the scarcity of datasets with hierarchical and fine-grained temporal annotations, we demonstrate that LLMs can effectively group atomic actions into higher-level events, enriching existing datasets. We then propose OpenHOUSE (Open-ended Hierarchical Online Understanding System for Events), which extends streaming action perception beyond action classification. OpenHOUSE features a specialized streaming module that accurately detects boundaries between closely adjacent actions, nearly doubling the performance of direct extensions of existing methods. We envision the future of streaming action perception in the integration of powerful generative models, with OpenHOUSE representing a key step in that direction.
Abstract (translated)
我们介绍了层次化流媒体视频理解任务，该任务结合了在线时序动作定位与自由形式描述生成。鉴于缺少具有分层和细粒度时间注释的数据集，我们展示了大型语言模型（LLMs）可以有效地将原子动作组合成更高层次的事件，从而丰富现有的数据集。然后，我们提出了OpenHOUSE（开放式层次化在线理解系统），该系统将流媒体动作感知扩展到了超出动作分类的范围。OpenHOUSE具有一个专门设计的流模块，能够准确检测紧密相邻的动作之间的边界，这几乎使现有方法直接扩展后的性能翻了一番。我们展望了流式动作感知的未来发展方向是集成强大的生成模型，并认为OpenHOUSE代表了朝这个方向迈出的关键一步。
URL
https://arxiv.org/abs/2509.12145
