video-ma2mba-0.3b

Video-Ma²mba is an efficient long-form video understanding model that replaces Transformer attention with State Space Models (SSMs) in the Mamba-2 framework, achieving linear time and memory scaling in sequence length. A single GPU can therefore process video sequences equivalent to millions of tokens — over two hours of video at 1 FPS.

Paper: Look Every Frame All at Once: Video-Ma²mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing

Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro

Integrated Vision and Language Lab, KAIST

What is Video-Ma²mba?

Processing long videos with Transformer-based video-LLMs is expensive: self-attention scales quadratically with the number of tokens, so memory and compute explode as videos grow to thousands of frames. Video-Ma²mba addresses this at the architecture level.

• State-space backbone. The language model is built on Mamba-2, replacing attention with selective state-space layers whose cost grows linearly with sequence length. A long video is consumed as one continuous stream rather than a truncated window.
• Multi-Axis Gradient Checkpointing (MA-GC). A training-time memory technique that checkpoints activations along multiple axes, sharply reducing the memory footprint versus standard gradient checkpointing and making long-sequence training feasible.
• Streaming-scale context. Together these let the model ingest sequences equivalent to millions of tokens — 2+ hours of video at 1 FPS — on a single GPU.

Architecture

Video (sampled at 1 FPS)
  │
  ▼
[SigLIP-so400m Vision Encoder]  →  per-frame visual tokens
  │
  ▼
[MLP Projector]                 →  language-model embedding space
  │
  ▼
[Mamba-2 Language Model]        ←  linear-time SSM · 48 layers · d=1024
  │
  ▼
Text response

Component	Specification
Architecture	LlavaMambaForCausalLM (llava_mamba)
Language model	Mamba-2 · 48 layers · hidden size 1024 · ~370M params
Vision encoder	google/siglip-so400m-patch14-384
Projector	2-layer MLP with GELU
Precision	bfloat16
Sampling	1 FPS with time-instruction conditioning
Max context	up to ~1M tokens

Results

Scores are as reported in the paper for the Video-Ma²mba family. The row marked (this) corresponds to this checkpoint.

Video-MME (w/o subtitles)

Model	Total Size	Short	Medium	Long	Overall
Video-Ma²mba-0.3B (this)	0.7B	37.4	35.0	26.8	33.1
Video-Ma²mba-1.3B	1.8B	49.4	39.2	31.9	40.3
Video-Ma²mba-2.7B	3.1B	57.6	42.7	35.4	45.2

LongVideoBench

Model	Total Size	Val	Test
Video-Ma²mba-0.3B (this)	0.7B	34.0	34.2
Video-Ma²mba-1.3B	1.8B	38.0	39.8
Video-Ma²mba-2.7B	3.1B	43.0	44.2

General Video Understanding

Model	Total Size	ActivityNet-QA	Video-ChatGPT	MVBench
Video-Ma²mba-0.3B (this)	0.7B	43.8	2.69	41.1
Video-Ma²mba-1.3B	1.8B	50.0	2.76	44.4
Video-Ma²mba-2.7B	3.1B	51.7	3.03	48.3

Model Variants

Model	LLM (Mamba-2)	Total
video-ma2mba-0.3b (this)	370M	0.7B
video-ma2mba-1.3b	1.3B	1.8B
video-ma2mba-2.7b	2.7B	3.1B

Usage

Video-Ma²mba uses a custom LlavaMambaForCausalLM architecture. For the inference pipeline, video preprocessing, and evaluation scripts, please refer to the official Video-Ma²mba GitHub repository.

Citation

@article{lee2024videoma2mba,
  title={Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing},
  author={Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
  journal={arXiv preprint arXiv:2411.19460},
  year={2024}
}

License

This model is released under the Apache 2.0 License.