Streaming VLM: Real-Time Understanding for Infinite Video Streams

100%

STREAMING VLM: REAL-TIME UNDERSTANDING FOR INFINITE VIDEO STREAMS

ABSTRACT

Vision-language models could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, Streaming VLM achieves a sixty-six point one eight percent win rate against GPT-four O mini and maintains stable, real-time performance at up to eight frames per second on a single NVIDIA H one hundred. Notably, our supervised fine-tuning strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on Long VideoBench by plus four point three zero and OVOBench Realtime by plus five point nine six.

One INTRODUCTION

Vision-language models could power autonomous driving, embodied agents, and real-time assistants, but they face critical challenges: understanding near-infinite video, responding in real time stably. To accept infinite input, common ideas are Sliding Window Attention with or without overlapping. As shown in Figure one: (a) Full Attention suffers from heavy memory and latency; (b) Sliding Window without Overlapping resets context frequently and breaks coherence; (c) Sliding Window Attention with Overlapping keeps recent tokens but recomputes attention many times, which hurts efficiency.

Aligning training with inference adds further challenges. Real streaming requires taking infinite visual input in real time and replying with very low delay, but training cannot use extremely long videos. Current approaches to KV cache eviction often lack alignment with the training phase. How to train on short videos and still enable the model to reason over very long streams remains underexplored. This leads to our core question: How can we train vision-language models to understand video chunks in real time and reason stably over infinite video, moving toward human-like intelligence?

In this paper, we propose Streaming VLM, a unified framework that aligns training with streaming inference and a dataset curation pipeline. The key ideas are: (one) Train the vision-language model with full attention on short, overlapped video chunks. (two) At inference, use an attention sink and a sliding window with to handle infinite video, aligned with training. (three) Reuse past KV states and use contiguous position IDs to keep inference stable.

Using this framework, we build Inf-Streams-Train, a sports commentary supervised fine-tuning dataset of over four thousand hours and Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. Then, we fine-tune Qwen-two point five-VL-seven B-Instruct for real-time commentary, yielding Streaming VLM that can understand infinite video and respond in real time. We evaluate Streaming VLM on captioning and VQA tasks, including LiveCC-Sports-three K CC and Inf-Streams-Eval for captioning, and Long VideoBench for video understanding.

On captioning tasks, Streaming VLM, with its infinite video understanding, outperforms existing models such as Livecc-seven B-Instruct. As shown in Figure two, Streaming VLM performs well on practical tasks: it can provide continuous commentary for more than two hours on sports games. On VQA tasks, even without any VQA fine-tuning, Streaming VLM still improves on Long VideoBench by plus four point three zero. In terms of efficiency, Streaming VLM maintains a low and stable latency, making it highly suitable for real-world streaming understanding tasks.

Two METHOD

Two point one INFERENCE SCHEME OF STREAMING VLM

Two point two TRAINING STRATEGY

Two point three. Data Curation Pipeline

Two point three point two. Data Cleaning

Two point three point three. SFT and Evaluation Data Segmentation

Two point three point four. High-Quality Annealing Data

Three. Experiments

Three point one. Experimental Setup

Three point two ACCURACY RESULTS

Three point two point two VQA

Three point three EFFICIENCY TESTS

Three point four ABLATION STUDY

Three point four point two SLIDING WINDOW AND SINK

Three point four point three TRAINING STRATEGY AND DATASET

Four RELATED WORK

Five CONCLUSION

A point one LLM USAGE STATEMENT

A point two STABILITY OVER TIME

A point three DEMO

Overview

The paper introduces Streaming VLM, a novel approach to real-time video understanding that combines effective training methods with fast inference techniques. It aims to maintain stable performance while processing nearly infinite streams of video data.

Key Points

1Streaming VLM effectively reduces computational costs associated with traditional attention mechanisms
2The model uses a compact KV cache to stabilize memory usage during streaming inference
3New evaluation benchmarks demonstrate Streaming VLM's performance compared to existing models
4The training strategy involves aligned, short video chunks to prepare for real-time analysis of continuous video input
5The model shows significant improvements in video question answering without requiring additional fine-tuning.

Details

Authors: Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, Song Han
Category: Technology and Engineering

PDF
Unit 1- Regression with Tidymodels
This unit covers the application of regression techniques using the tidymodels ecosystem in R, including linear and logistic regression, model fitting, evaluation, and different machine learning methods under predictive and prescriptive analytics.
PDF
Artificial Intelligence as a general-purpose technology: an historical perspective
This paper examines the impact of general-purpose technologies like artificial intelligence on productivity, drawing historical parallels with previous technological revolutions to explore potential future implications for economic growth.
PDF
Structured fabrics with tunable mechanical properties
This article discusses the development of structured fabrics with tunable mechanical properties, focusing on their design, experimental validation, and potential applications in wearable technology and adaptive materials.
PDF
Specification Engineering: The New Skill After Prompt Engineering
This document discusses the emerging skill of specification engineering, which goes beyond traditional prompt engineering to define the goals, constraints, and validation processes for AI-assisted tasks. It highlights the importance of clear specifications for improved outcomes in various AI applications.
PDF
AD95 Information Assurance and IT Security
This document outlines the information assurance and IT security policies at Penn State University, detailing responsibilities, guiding principles, classification of information, and mandatory adherence to security standards.