wpj20000/DeltaV-2B

DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models

A unified large multimodal model that thinks with visual state updates—modeling only the sparse, reasoning-critical changes across reasoning steps instead of regenerating full images.

---

News

• 2026.07.09 🚀 We release DeltaV-2B, a unified large multimodal model for interleaved multimodal reasoning.

Introduction

DeltaV is a unified large multimodal model (ULMM) designed to think with visual state updates during interleaved multimodal reasoning. Conditioned on historical visual states, it incrementally predicts compact visual update tokens that capture sparse but reasoning-critical changes across reasoning steps, avoiding repeated modeling of unchanged content. Token budgets are dynamically allocated by the TSIM Router according to temporal visual variation, and visual states are encoded by the TSIM-Tok tokenizer.

This repository releases the DeltaV-2B ULMM together with the TSIM-Tok tokenizer, inference scripts, and tiny samples.

TODO / Roadmap

This release focuses on inference with the pretrained DeltaV-2B checkpoint. The following components are not included yet and will be released in a future update:

• StructCoT dataset — the full StructCoT dataset and training data. (A small inference example, data/struct_infer_sample.json, is included so inference is runnable; the benchmark numbers in the tables below are kept as a record.)
• Zebra-CoT / StructCoT evaluation (scoring) tools — the LLM-API-based scorers and their guide, pending the public StructCoT release.
• DeltaV training — the two-stage DeltaV ULMM training scripts and configs.
• TSIM-Tok training and testing — the TSIM-Tok tokenizer training and reconstruction-evaluation code (the tokenizer model is retained as DeltaV's visual backbone).

DeltaV Workflow

https://github.com/user-attachments/assets/724b2ff7-279a-4139-a9a0-de734014431d

Repository Layout

deltav/        DeltaV model code: modeling, processing, configuration, backbone
  tsim_tok/  TSIM-Tok visual tokenizer and TSIM Router
inference/   Inference
scripts/     Ready-to-run scripts for DeltaV inference and data utilities
configs/     Model and acceleration configs
data/        Tiny samples
docs/        Extended tutorials and README media assets
tools/       Data processing and inference post-processing

Installation

See INSTALL.md for the full setup guide. Quick version:

conda create -n deltav python=3.10 -y && conda activate deltav
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Download Checkpoints

Download our models from Huggingface.

pip install huggingface_hub

python tools/download_model.py -n DeltaV-2B      # DeltaV-2B (legacy CLI identifier)

You can also download our models from ModelScope.

pip install modelscope

python tools/download_model.py -t modelscope -n DeltaV-2B   # DeltaV-2B (legacy CLI identifier)

The released checkpoint is placed under weights/:

weights/
  deltav_2b/

Inference

DeltaV training is not included in this release. The two-stage DeltaV ULMM training recipe (on top of a frozen TSIM-Tok) will be added back later — see TODO / Roadmap.

DeltaV Inference

# Pure inference for evaluation. Outputs text-only .json, then merge + extract answers.
MODEL_PATH=weights/deltav_2b \
JSON_PATH=data/zebra_infer_sample.json \
bash scripts/deltav/infer_deltav.sh

To also decode and save generated images, set VIS_ARGS. This streams results to .jsonl and skips merge/extract steps. Add --concat_gt_images to dump a ground-truth montage alongside each prediction.

MODEL_PATH=weights/deltav_2b \
JSON_PATH=data/zebra_infer_sample.json \
VIS_ARGS="--decode_and_save_image --concat_gt_images" \
bash scripts/deltav/infer_deltav.sh

Input format & token budgets

Each inference sample carries the prompt, its input/output image paths, and (optionally) the per-image incremental token budget:

{
  "config": "Visual Logic & Strategic Games - Tetris",
  "input_prompt": "Fill the entire grid EXCEPT ...",
  "input_image":  ["/abs/path/problem.jpg"],
  "output_image": ["/abs/path/reasoning_01.jpg", "..."],
  "num_tokens": [144, 100, 81, ...]
}

num_tokens is the per-image incremental token budget: the first image always uses the base budget n_base; each subsequent image uses a routed budget. There are two ways to supply it:

• Precomputed (default) — if samples already carry num_tokens, pass --use_json_num_tokens (this is the default TOKEN_ARGS in infer_deltav.sh).
• TSIM Router (on the fly) — otherwise set TOKEN_ARGS="--use_tsim_router --visual_extractor_repo <dinov2> --visual_extractor_ckpt <dinov2.pth>". The router (deltav/tsim_tok/tsim_router.py) measures temporal visual change with a frozen DINOv2 ViT-B/14 and maps it to budgets via tools/data_processing/tsim_intervals.json. DINOv2 is fetched automatically by torch.hub on first use; to run offline, clone it and pass the local paths (see INSTALL.md).

TSIM-Tok training and reconstruction evaluation are not included in this release. The tokenizer model is retained as DeltaV's visual backbone, but its training/testing code will be added back later — see TODO / Roadmap.

Documentation

• docs/eval_vlmevalkit.md: understanding benchmarks via VLMEvalKit. This guide is still being refined.

The following guides will be published alongside the training / evaluation code (see TODO / Roadmap):

• docs/data_and_token.md: dataset format and the offline TSIM Router pipeline that turns image similarity into per-image token budgets.
• docs/eval_zebra_struct.md: Zebra-CoT and StructCoT scoring.
• docs/advanced_zero3_gc.md: ZeRO-3 and gradient checkpointing.
• docs/packing.md: sequence packing, length computation, and packing training.

Qualitative Examples

Qualitative comparison of multimodal reasoning. Full-image modeling (Base) exhibits inconsistent intermediate visual states, while DeltaV maintains consistent visual representations through visual state updates.

Benchmark

External Multimodal Reasoning and Understanding Evaluation

Model	#Param	VStar	EMMA	M3CoT	MathVista	VisuLogic	MMBench	MME‑P	MMVP
General ULMMs
Chameleon	7B	32.5	8.6	16.1	21.7	4.5	6.0	530	4.7
Anole	7B	34.0	6.6	15.8	22.5	3.7	6.2	508	6.7
Janus-pro	1B	43.5	18.9	45.9	37.6	25.0	60.2	1398	39.3
Janus-pro	7B	39.3	21.5	49.1	42.7	17.5	66.7	1509	34.7
OmniGen2	7B	41.4	14.7	50.3	60.2	0.1	76.1	1588	35.3
Bagel	7B	70.1	28.7	31.4	72.5	28.9	83.7	1665	69.3
EMU3.5	34B	-	-	-	28.3	11.4	13.7	791	16.7
Understanding-centric MLLMs
Qwen3-VL	2B	71.7	22.2	53.0	61.1	11.5	77.1	1482	45.0
Qwen3-VL	8B	83.7	30.6	61.2	77.6	22.5	85.2	1729	59.3
InternVL3.5	2B	68.1	12.7	51.3	60.8	26.0	78.2	1552	48.7
InternVL3.5	8B	69.1	16.6	59.9	74.1	29.7	82.7	1688	57.3
Latent Interleaved Reasoning Models
Monet	7B	79.1	22.1	44.2	62.5	10.6	75.3	1636	48.7
Mirage	8B	13.6	13.9	1.08	29.9	0.4	12.3	549	0.0
VPT-Det	2B	43.5	20.1	44.4	41.8	25.6	73.3	1516	34.0
Explicit Interleaved Reasoning ULMMs
Bagel-Zebra-CoT	7B	64.9	20.6	62.6	72.1	0	55.6	1647	22.0
ThinkMorph	7B	64.4	22.4	48.8	67.8	6.5	78.2	1478	8.6
DeltaV [Weight]	2B	75.9	28.6	54.5	69.3	23.5	82.3	1555	51.3

VStar, EMMA, M3CoT, MathVista, and VisuLogic are grouped as multimodal reasoning benchmarks, while MMBench, MME‑P, and MMVP are grouped as multimodal understanding benchmarks.

In-domain Multimodal Reasoning Evaluation

Model	#Param	Zebra-CoT					StructCoT
		2D	3D	Science	Strategy	Overall	Strategy Planning	Spatial Planning	Logic	Math	Science	Visual Search	Jigsaw Restoration	Overall
Understanding-centric MLLMs
GPT-5.2	-	67.6	19.3	73.3	54.4	53.7	43.1	33.8	42.1	76.3	50.4	87.0	57.1	55.7
Gemini-3.1 Pro	-	68.7	19.0	83.3	60.4	57.9	71.6	28.2	50.2	78.3	55.0	79.4	65.3	61.1
Gemini 3.0 Flash	-	66.5	19.4	78.4	54.5	54.7	55.0	33.3	44.8	74.8	48.4	83.6	64.9	57.8
Qwen3-VL	2B	44.3	13.2	30.3	9.2	24.3	3.4	31.4	4.6	41.4	29.4	80.8	39.3	32.9
Qwen3-VL	8B	50.7	16.9	56.0	22.7	36.6	21.6	25.4	13.1	59.3	39.3	83.8	46.5	41.3
InternVL3.5	8B	29.7	11.4	48.9	19.8	27.5	6.9	36.3	17.5	36.1	32.0	75.8	41.0	35.1
Qwen2.5-VL	72B	43.2	17.3	50.1	25.8	34.1	14.8	34.4	31.4	48.0	36.5	84.9	47.0	42.4
General ULMMs
Chameleon	7B	13.3	3.0	5.2	9.9	7.9	5.6	12.5	4.1	9.1	13.1	23.5	14.4	11.8
Anole	7B	10.8	2.8	4.8	8.5	6.7	5.4	0.1	3.8	8.9	12.8	16.8	11.4	9.9
Janus-pro	7B	31.7	7.7	11.5	18.0	17.2	4.3	24.4	13.4	16.6	12.0	74.6	33.9	25.6
OmniGen2	7B	26.5	1.3	9.6	9.7	11.8	0.6	25.3	1.5	8.4	10.1	78.1	28.5	21.8
Bagel	7B	43.3	14.7	44.5	16.3	29.7	16.4	24.9	12.8	49.0	35.5	84.6	49.0	38.9
EMU3.5	34B	10.1	3.6	8.6	11.8	8.5	2.8	29.1	4.6	19.3	15.6	21.1	18.8	15.9
Latent Interleaved Reasoning Models
Monet	7B	37.5	12.0	15.1	23.0	21.9	2.3	19.9	21.9	33.8	25.8	59.6	33.8	28.1
Mirage	8B	2.2	2.5	10.7	12.4	7.0	0.9	14.3	12.4	35.8	22.5	11.0	30.4	18.2
VPT-Det	2B	32.3	3.5	6.5	18.7	15.3	7.5	26.5	8.8	14.5	15.1	73.1	35.9	25.9
Explicit Interleaved Reasoning ULMMs
Bagel-Zebra-CoT	7B	-	-	-	-	-	7.0	24.6	22.8	33.3	27.3	81.0	41.9	34.0
ThinkMorph	7B	43.0	11.6	31.4	22.9	27.2	21.4	19.5	26.4	43.4	26.0	84.1	49.9	38.7
DeltaV [Weight]	2B	78.9	20.0	41.1	38.3	44.6	16.4	53.0	66.0	30.1	45.6	84.3	62.6	51.1

The StructCoT test set excludes all samples originating from the Zebra-CoT dataset.

Acknowledgements

We would like to thank Qwen3-VL and VFMTok for providing base models and code, as well as their contributions to this field. We also thank Zebra-CoT for providing a valuable interleaved multimodal reasoning dataset. We also thank everyone who contributed to this open-source effort.

Copyright

Please do not hesitate to share your valuable feedback—it is a key motivation that drives us to continuously improve our framework.

Note: Our model is intended for academic research and non-commercial use only. If you are interested in a faster (smaller) or stronger model, please contact us at xbai@hust.edu.cn or ylliu@hust.edu.cn.