TimeChat-Captioner:
Scripting Multi-Scene Videos with Time-Aware
and Structural Audio-Visual Captions

Linli Yao1,2, Yuancheng Wei3, Yaojie Zhang4, Lei Li5, Xinlong Chen6,2,
Feifan Song1, Ziyue Wang1, Kun Ouyang1, Yuanxin Liu1, Lingpeng Kong5,
Qi Liu5, Pengfei Wan2, Kun Gai2, Yuanxing Zhang2, Xu Sun1,†
1Peking University, 2Kling Team, Kuaishou Technology, 3South China University of Technology,
4University of Electronic Science and Technology of China, 5The University of Hong Kong, 6Institute of Automation, Chinese Academy of Sciences
†Corresponding Author
Paper Code & Data Benchmark Dataset Model
ICML 2026
Scroll Down

Interactive Demo

💡 Tip: Select a segment below to view detailed annotations
00:00-00:10

Detailed Events

00:11-00:20

Detailed Events

00:21-00:28

Detailed Events

00:29-00:36

Detailed Events

00:37-00:49

Detailed Events

00:50-00:59

Detailed Events

Abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. The six dimensions span Detailed Events, Visual Background, Camera State, Shot Editing Style, Dialogue Content, and Acoustics Content.

To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark of 1,122 videos averaging 995 words per video, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity via IoU-based Dynamic Programming alignment and many-to-one merging.

Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline built on Qwen2.5-Omni and trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance with a SodaM score of 35.0, surpassing Gemini-2.5-Pro (33.7), while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are publicly available.

Time-Aware and Structural Audio-Visual Captions

TimeChat-Captioner generates script-like, temporally dense, and structurally rich audio-visual narratives for multi-scene videos. At its core is the innovative six-dimensional structural schema that comprehensively describes video content like a professional film script.

  • Detailed Events: Narrating audiovisual content and actions
  • Visual Background: Depicting setting, location, and atmosphere
  • Camera State: Describing camera movements, angles, and framing
  • Shot Editing Style: Analyzing post-production editing techniques
  • Dialogue Content: Transcription and summary with speaker identification
  • Acoustics Content: Portraying background sounds, music, and speaking tones
TimeChat-Captioner Task Overview

TimeChat-Captioner generates temporally-dense and description-dense captions with six structural dimensions.

SodaM: A Unified Evaluation Metric

Evaluating continuous captions faces the "boundary ambiguity" challenge—scene boundaries are subjective, and predictions may be more fine-grained than ground truth. We propose SodaM, a unified metric that addresses these challenges through a sophisticated two-stage alignment strategy.

How SodaM Works

  1. IoU-based Dynamic Programming Alignment: Uses Dynamic Programming to find the optimal alignment path through the (M, N) scene grid based on temporal Intersection over Union (IoU), ensuring the best match between predicted and ground-truth scene boundaries.
  2. Many-to-One Merging: Gracefully handles cases where models generate finer-grained segments than human references by concatenating predicted captions for a single ground-truth match, avoiding unfair penalties for detailed predictions.
  3. CheckList Score for Semantic Completeness: Uses an LLM (Gemini-2.5-Flash) to judge if predicted captions cover atomic elements (keypoints) from the ground truth, providing a holistic measure of semantic coverage across all six dimensions.

TimeChat-Captioner-7B Architecture

TimeChat-Captioner-7B is built on Qwen2.5-Omni with key architectural innovations for precise temporal localization and synchronous audio-visual perception.

Composite Reward Function

Format Reward

Binary reward for valid JSON-formatted output (1 if parseable, 0 otherwise).

Length Reward

Regularizes output length to prevent hallucinations and repetitive content.

Timestamp Reward

Average F1 score at IoU thresholds {0.3, 0.5, 0.7, 0.9} for temporal accuracy.

Time-aware Caption Reward

Uses the SodaM metric to measure semantic completeness and temporal alignment across all six dimensions.

TimeChat-Captioner-7B Architecture

The architecture of TimeChat-Captioner-7B with synchronous audio-visual perception and M-RoPE for temporal localization, jointly optimized by the composite reward above.

Key Architectural Features

  • Synchronous Perception:

    Interleaved audio and visual tokens in a single sequence, enabling holistic audio-visual understanding without separate processing streams.

  • Multimodal RoPE (M-RoPE):

    Multimodal Rotary Position Embedding that encodes precise absolute temporal information, crucial for generating accurate MM:SS timestamps in captions.

  • Two-Stage Training (SFT + GRPO):

    Stage 1: Supervised Fine-Tuning on TimeChatCap-42K teaches the complex "script" format. Stage 2: Group Relative Policy Optimization (GRPO) jointly optimizes temporal accuracy and caption quality through a composite reward function.

Dataset Construction

We construct two complementary datasets to support the dense video captioning task: OmniDCBench as a high-quality human-annotated benchmark and TimeChatCap-42K as a large-scale training set.

OmniDCBench: Human Benchmark

OmniDCBench is entirely human-annotated by experts with cinematography knowledge, ensuring high-quality annotations that capture the nuances of professional video production.

OmniDCBench Statistics

  • Scale: 1,122 high-resolution video clips
  • Annotation Depth: Averaging 995 words per video
  • Expert Annotation: Entirely human-annotated by experts with cinematography knowledge
  • Coverage: All six structural dimensions
Statistics of human-annotated OmniDCBench

TimeChatCap-42K: Training Set

TimeChatCap-42K is synthesized using Gemini-2.5-Pro via a three-stage pipeline: (1) Boundary Segmentation for establishing rough timestamps, (2) Detailed Caption Generation for expanding into the six-dimensional structural schema, and (3) Quality Filtering to ensure high-quality annotations.

TimeChatCap-42K Statistics

  • Scale: 42,000 high-quality video-script pairs
  • Synthesis Pipeline: Three-stage process using Gemini-2.5-Pro
  • Quality: Structured captions following the six-dimensional schema
  • Purpose: Large-scale training for teaching the model complex "script" format
Statistics of the training dataset TimeChatCap-42K
Overview of the synthetic training data construction pipeline

Overview of the synthetic training data construction pipeline for TimeChatCap-42K.

Experimental Results

Main Results on OmniDCBench

TimeChat-Captioner-7B achieves a SodaM score of 35.0, surpassing Gemini-2.5-Pro (33.7) and Gemini-2.5-Flash (30.0), establishing state-of-the-art performance. The model significantly outperforms previous open-source models like Qwen3-Omni (14.3) and MiniCPM-o-2.6 (5.4).

Main Results on OmniDCBench

Main experimental results on OmniDCBench. TimeChat-Captioner-7B shows particular strength in Camera (12.4), Acoustics (38.2), and Dialogue (54.3) dimensions.

Downstream Generalization: Caption-based Audio-Visual Reasoning

We probe the information richness of generated captions with a caption-based QA protocol: each model produces video captions, and a frozen judge LLM answers Omni Video QA questions using only those captions as evidence—no video access. Higher QA accuracy therefore reflects more faithful and complete audio-visual content packed into the caption. Under this protocol, captions from TimeChat-Captioner-7B drive the judge to 52.8 on DailyOmni and 22.6 on WorldSense (best among open-source models), demonstrating that our time-aware, six-dimensional captions transfer rich audio-visual evidence to downstream reasoning.

Caption-based Audio-Visual Reasoning Results

Caption-based QA accuracy on DailyOmni and WorldSense. The judge LLM answers each Omni Video QA item using only the caption produced by each captioner (no raw video), so the score reflects information richness of the caption itself.

Temporal Grounding on Charades-STA

On the temporal grounding task Charades-STA, TimeChat-Captioner-7B achieves an R1@0.7 of 48.3, outperforming specialized expert models like TimeSuite (43.0). This demonstrates that precise temporal localization capabilities learned through M-RoPE and GRPO training generalize effectively to temporal grounding tasks.

Temporal Grounding Results

Results on Charades-STA temporal grounding benchmark, showing superior performance compared to specialized models.

BibTeX

@inproceedings{yao2026timechatcap,
  title={TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions},
  author={Yao, Linli and Wei, Yuancheng and Zhang, Yaojie and Li, Lei and Chen, Xinlong and Song, Feifan and Wang, Ziyue and Ouyang, Kun and Liu, Yuanxin and Kong, Lingpeng and others},
  booktitle={Proceedings of the Forty-Third International Conference on Machine Learning},
  year={2026}
}

Acknowledgement

Usage and License Notices: The data, code and checkpoints are intended and licensed for research use only. They are also restricted to uses that follow the license agreements of the respective datasets and models used in this work.

Related Projects: TimeChat-Online, TimeChat, Qwen2.5-Omni, Gemini