arxiv:2611.01120

StreamDuo: A Live Commentator and a Research Agent for Unseen Games

Published on Oct 31

· Submitted by

Penghao Luo on Nov 5

Alaya Lab

Upvote

Authors:

Xiangcheng Cao ,

Zhen Li ,

Abstract

StreamDuo pairs a trainable commentator with a frozen research agent that retrieves game wiki entries to improve cross-game streaming commentary without training on target games.

Generated by thinkingmachines/Inkling-Small

Streaming video LLMs can now commentate live gameplay, but they are trained and tested on the same games. We study cross-game streaming commentary, in which a model trained on one game commentates games it has never seen, and find that current commentators transfer their timing but not their knowledge of the game world. We propose StreamDuo, which pairs a trainable commentator with a frozen research agent that reads the wiki of the game being played. When the commentator sees something it cannot name, it sends the current frames, and the research agent returns a few wiki entries two seconds later while the stream continues. We train only the commentator, on a single game, in two stages. Information-gain labels teach it to ask only where an answer would help it say what the human caster said, and on-policy self-distillation from a copy of itself that sees the ideal answer teaches it to use noisy answers. Trained on Baldur's Gate 3 and tested on ten unseen games, StreamDuo raises entity recall from 0.082 to 0.116 over Proact-VL trained on the same data, cuts source-game leakage from 9.9% to 1.0%, and achieves the best timing and the highest judged liveliness and overall quality among streaming models.

View arXiv page View PDF Project page GitHub 11 Add to collection

Community

imkenLuo

Paper submitter 3 days ago

🎮 Can a live AI commentator handle a game it has never seen before?

We introduce StreamDuo, a two-agent framework for cross-game streaming commentary. Instead of relying solely on memorized game knowledge, a live commentator selectively queries a frozen research agent that retrieves relevant information from game wikis, all while the stream continues.

Our key contributions include:

Information-gain-based querying to learn when external knowledge is actually needed.
On-policy self-distillation to help the commentator use noisy retrieved information effectively.
Cross-game evaluation on 10 unseen games after training on only Baldur's Gate 3.

StreamDuo improves entity recall from 0.082 to 0.116 and reduces source-game knowledge leakage from 9.9% to 1.0% compared with the retrained Proact-VL baseline, while achieving stronger commentary timing and quality.

We welcome feedback and discussions!

imkenLuo

Paper submitter 2 days ago

Excited to share StreamDuo: A Live Commentator and a Research Agent for Unseen Games! 🎮

Can a live game commentator generalize to games it has never seen before?

We introduce StreamDuo, a dual-agent framework that combines a streaming video commentator with a knowledge-retrieval research agent to enable real-time commentary across unseen games.

Our key contributions:

Information-Gain-Guided Asking: Learn when external knowledge is actually useful, avoiding unnecessary retrieval.
On-Policy Self-Distillation: Train the commentator to use noisy retrieved knowledge more effectively.
Cross-Game Generalization: Evaluate on 10 unseen games, improving entity grounding and reducing source-game knowledge leakage.

🔗 Project page: https://zixiangao.github.io/StreamDuo/

💻 Code: https://github.com/ZixianGao/StreamDuo-code

We welcome feedback and discussions!