Streaming VLM: Real-Time Understanding for Infinite Video Streams

100%

STREAMING VLM: REAL-TIME UNDERSTANDING FOR INFINITE VIDEO STREAMS

ABSTRACT

Vision-language models could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, Streaming VLM achieves a sixty-six point one eight percent win rate against GPT-four O mini and maintains stable, real-time performance at up to eight frames per second on a single NVIDIA H one hundred. Notably, our supervised fine-tuning strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on Long VideoBench by plus four point three zero and OVOBench Realtime by plus five point nine six.

One INTRODUCTION

Vision-language models could power autonomous driving, embodied agents, and real-time assistants, but they face critical challenges: understanding near-infinite video, responding in real time stably. To accept infinite input, common ideas are Sliding Window Attention with or without overlapping. As shown in Figure one: (a) Full Attention suffers from heavy memory and latency; (b) Sliding Window without Overlapping resets context frequently and breaks coherence; (c) Sliding Window Attention with Overlapping keeps recent tokens but recomputes attention many times, which hurts efficiency.

Aligning training with inference adds further challenges. Real streaming requires taking infinite visual input in real time and replying with very low delay, but training cannot use extremely long videos. Current approaches to KV cache eviction often lack alignment with the training phase. How to train on short videos and still enable the model to reason over very long streams remains underexplored. This leads to our core question: How can we train vision-language models to understand video chunks in real time and reason stably over infinite video, moving toward human-like intelligence?

In this paper, we propose Streaming VLM, a unified framework that aligns training with streaming inference and a dataset curation pipeline. The key ideas are: (one) Train the vision-language model with full attention on short, overlapped video chunks. (two) At inference, use an attention sink and a sliding window with to handle infinite video, aligned with training. (three) Reuse past KV states and use contiguous position IDs to keep inference stable.

Using this framework, we build Inf-Streams-Train, a sports commentary supervised fine-tuning dataset of over four thousand hours and Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. Then, we fine-tune Qwen-two point five-VL-seven B-Instruct for real-time commentary, yielding Streaming VLM that can understand infinite video and respond in real time. We evaluate Streaming VLM on captioning and VQA tasks, including LiveCC-Sports-three K CC and Inf-Streams-Eval for captioning, and Long VideoBench for video understanding.

On captioning tasks, Streaming VLM, with its infinite video understanding, outperforms existing models such as Livecc-seven B-Instruct. As shown in Figure two, Streaming VLM performs well on practical tasks: it can provide continuous commentary for more than two hours on sports games. On VQA tasks, even without any VQA fine-tuning, Streaming VLM still improves on Long VideoBench by plus four point three zero. In terms of efficiency, Streaming VLM maintains a low and stable latency, making it highly suitable for real-world streaming understanding tasks.

Two METHOD

Two point one INFERENCE SCHEME OF STREAMING VLM

Two point two TRAINING STRATEGY

Two point three. Data Curation Pipeline

Two point three point two. Data Cleaning

Two point three point three. SFT and Evaluation Data Segmentation

Two point three point four. High-Quality Annealing Data

Three. Experiments

Three point one. Experimental Setup

Three point two ACCURACY RESULTS

Three point two point two VQA

Three point three EFFICIENCY TESTS

Three point four ABLATION STUDY

Three point four point two SLIDING WINDOW AND SINK

Three point four point three TRAINING STRATEGY AND DATASET

Four RELATED WORK

Five CONCLUSION

A point one LLM USAGE STATEMENT

A point two STABILITY OVER TIME

A point three DEMO

Overview

The paper introduces Streaming VLM, a novel approach to real-time video understanding that combines effective training methods with fast inference techniques. It aims to maintain stable performance while processing nearly infinite streams of video data.

Key Points

1Streaming VLM effectively reduces computational costs associated with traditional attention mechanisms
2The model uses a compact KV cache to stabilize memory usage during streaming inference
3New evaluation benchmarks demonstrate Streaming VLM's performance compared to existing models
4The training strategy involves aligned, short video chunks to prepare for real-time analysis of continuous video input
5The model shows significant improvements in video question answering without requiring additional fine-tuning.

Details

Authors: Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, Song Han
Category: Technology and Engineering

PDF
CENGR 2220: Geology for Civil Engineers - Structural Geology and Rock Mechanics
This document is a lesson plan for a geology course aimed at civil engineering students, focusing on the concepts of structural geology, including attitudes of beds, types of outcrops, and geological maps.
PDF
Operating Systems
This document provides an overview of operating systems, including their functions, types, and the essential role they play in managing computer hardware and software resources. It outlines the basic operations of an operating system and explains how they serve as an interface between the user and computer hardware.
PDF
Force, Motion, and Energy: Comprehensive Guide to Mechanics and Electricity
This document provides a detailed overview of fundamental concepts in mechanics and electricity, covering topics such as the laws of motion, force, energy, and electrical circuits. It illustrates these principles with practical examples and problems, making it suitable for educational purposes.
PDF
Introduction to Word Processing
This document provides an overview of word processing, its historical development, and the primary functions of software like Microsoft Word. It discusses key features, objectives for users, and various applications in business, education, and creative writing.
PDF
Wheel-INS2: Multiple MEMS IMU-based Dead Reckoning System with Different Configurations for Wheeled Robots
This document presents a multiple IMUs-based dead reckoning solution for wheeled robots, improving navigation through the integration of IMUs mounted in various positions on the vehicle to enhance accuracy and robustness.