跳到正文
实时
Hugging Face Daily Papers·· 22 小时前

FastFrames:实现交互式延迟的视频生成模型

FastFrames: Video Generation at Interactive Latency

要点速览

FastFrames 是一组快速自回归视频扩散模型,在单块 B200 GPU 上以 36ms/帧(28RPS)的速度实现交互式延迟,同时匹配了近乎 8 倍慢速模型的生成质量;其 14B 版本在 8x B200 GPU 节点上以 45ms/帧(22RPS)运行,并显著提升视觉质量与真实感。

正文

Published on Nov 17

Authors:

,

,

,

,

,

Abstract

Autoregressive video diffusion models generate frames based on new observations and inputs, making them a promising medium for interactive visual experiences. However, existing models are orders of magnitude too slow for fluid interaction. This paper introduces FastFrames, a family of fast autoregressive video diffusion models that achieves interactive latency without compromising quality. We profile the inference pipeline of state-of-the-art models and find that operations often treated as fixed implementation details, such as VAE decoding and KV cache updates, in fact account for over half of end-to-end inference latency. We challenge the assumption that these components are indispensable and replace them with lightweight alternatives. Our 1.3B variant runs at 36 ms per frame, or 28 responses per second (RPS), on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8x slower. Our 14B variant runs at 45 ms per frame, or 22 RPS, on an 8x B200 GPU node and yields significantly better visual quality and realism. We show that FastFrames enables real-time interaction in applications such as camera-conditioned and physics-conditioned video generation.

View arXiv page View PDF Project page GitHub 14 Add to collection

来源: Hugging Face Daily Papers · huggingface.co