Gander

Streaming audio-visual interaction and agentic task orchestration

Gander is an omni-interaction model designed for continuous spoken and visual interaction while long-running tasks execute asynchronously. It combines a realtime multimodal Thinker, a streaming speech Talker, and the Gander runtime for task lifecycle management.

Model Overview

Gander organizes interaction into causal one-second units. In each unit, newly available audio and visual observations precede the model decision. The Thinker predicts whether to listen, speak, interrupt, or invoke a task operation, and produces text or structured calls when required. The Talker conditions on Thinker representations to synthesize speech incrementally.

The model supports:

• continuous speech interaction, interruption handling, and conversational backchannels;
• streaming image and video understanding with temporally grounded responses;
• structured task delegation, follow-up instructions, progress interaction, and completion;
• incremental speech generation synchronized with the Thinker output.

Components

Component	Description
thinker/	Multimodal perception, interaction control, language generation, and task operations
talker/	Streaming S3 speech-token generation and Token2wav waveform decoding
release_manifest.json	Model provenance and the shared temporal-unit contract

Model Specification

Property	Value
Base model	openbmb/MiniCPM-o-4_5
Input modalities	Text, audio, image, and video frames
Temporal unit	1 second
Thinker output budget	Up to 8 lexical tokens per speaking unit
Talker output budget	50 S3 tokens per speaking unit
Streaming speech chunk	25 S3 tokens, approximately 0.5 seconds
Audio input sample rate	16 kHz
Speech output sample rate	24 kHz
Model precision	BF16

Usage

Thinker

import torch
from transformers import AutoModel, AutoProcessor, AutoTokenizer

path = "/path/to/gander/thinker"

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModel.from_pretrained(
    path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=True,
    init_tts=False,
).eval()

Full Realtime Inference

The complete audio-visual and speech experience uses the Gander runtime. Start from its serving example and configure the base model together with both released components:

model:
  model_name_or_path: /path/to/MiniCPM-o-4_5
  processor_name_or_path: /path/to/MiniCPM-o-4_5
  init_vision: true
  init_audio: true
  init_tts: true
  token2wav_dir: /path/to/gander/talker/assets/token2wav

duplex:
  checkpoint: /path/to/gander/thinker
  talker_checkpoint: /path/to/gander/talker
  detached_talker_device: cuda:1
  ref_audio_path: /path/to/gander/talker/assets/ref_audio.wav
  speak_text_tokens_per_unit: 8
  talker_speech_tokens_per_unit: 50
  talker_emit_speech_tokens: 25

git clone https://github.com/Omni-Interaction-Gander/Omni-Interaction-Agent.git
cd Omni-Interaction-Agent
cp gander_runtime/configs/serve.example.yaml gander_runtime/configs/serve.local.yaml
# Apply the paths above to serve.local.yaml.
./scripts/serve.sh gander_runtime/configs/serve.local.yaml

This configuration loads the released Thinker and Talker as a matched Unit-8/50 pair and runs the Talker asynchronously on the second GPU.

Training

Both components are initialized from MiniCPM-o 4.5 and trained in two stages:

Stage	Trainable modules	Schedule
Thinker	Language model and audio projection	1 epoch, 8,407 optimization steps
Talker	Semantic projection and speech decoder	2 epochs, 3,246 optimization steps

Training uses a multimodal mixture spanning spoken dialogue, full-duplex interaction, streaming video understanding, and agent task lifecycles. The training representation follows the same causal one-second unit structure used during online inference.

Intended Use

Gander-Unit8 is intended for research on realtime multimodal assistants, full-duplex spoken interaction, streaming video-language modeling, and interactive agent systems. External task execution is coordinated by the Gander runtime and its configured worker provider.

Limitations

Model outputs may contain factual or perceptual errors, particularly under ambiguous audio, rapid visual changes, or long temporal dependencies. Applications should validate consequential external actions and follow deployment requirements appropriate to their domain.

Resources

Resource	Link
GitHub	Omni-Interaction-Agent
Paper	Arxiv Link
Demo	Project Page
Dataset	Coming soon

License

Gander-Unit8 is released under the Apache License 2.0 and is derived from openbmb/MiniCPM-o-4_5.