DreamX-Creator 1.0: Model Weights

DreamX Team

---

This directory holds all model weights for DreamX-Creator 1.0, a research framework for native joint audio-video generation. Given a first frame and a text prompt, the 7B base generator jointly models modality-specialized video and audio streams; the Autoregressive 1-Step 2K Refiner (SR-DiT 5B) then upgrades the generated video to high-quality 2K output.

Weights are not stored in the git repository. They are distributed on HuggingFace and ModelScope and should be placed under this directory following the layout below.

Expected Layout

checkpoints/
├── creator/                     # DreamX-Creator 1.0 joint generator (7B, LoRA merged)
│   ├── video_model/             # video DiT shards + config
│   ├── audio_model/             # audio DiT + config
│   └── cross_attn_weights.safetensors  # gated A2V/V2A cross-modal attention
├── audio_vae/                   # CreatorDACVAE audio VAE
├── refiner/                     # 2K refiner
│   ├── sr_dit_5b.pt             # SR-DiT 5B refiner
│   ├── latent_upsampler_flash.pt       # FlashLatentUpsampler (default)
│   ├── latent_upsampler_2d_causal.pt   # causal 2D latent upsampler (optional)
│   └── lightvae_nu_scheme3.pt          # distilled fast decoder (optional, off by default)
└── wan2.2_ti2v_5b/              # shared Wan2.2-TI2V-5B dependencies
    ├── Wan2.2_VAE.pth           # video VAE
    ├── models_t5_umt5-xxl-enc-bf16.pth  # UMT5-xxl text encoder
    └── google/umt5-xxl/         # tokenizer

The wan2.2_ti2v_5b/ directory can also be downloaded directly from Wan-AI/Wan2.2-TI2V-5B; only the three entries above are needed. It is shared by both pipelines.

Which Weights Are Used Where

• Joint audio-video generation uses creator/, audio_vae/, and wan2.2_ti2v_5b/. See the audio_video_generation README for setup, input overrides, and memory options.
• 2K refinement uses refiner/ and wan2.2_ti2v_5b/. See the video_refiner README for setup and the full list of inference knobs.

Quickstart

Once the weights above are in place, from the repository root:

1. Joint audio-video generation (first frame + prompt to synchronized video with audio):

cd audio_video_generation
pip install -r requirements.txt
./inference.sh                        # uses assets/case1.png by default

2. 2K refinement (super-resolve a generated or external video, audio unchanged):

cd ../video_refiner
pip install -r requirements.txt
INPUT=/path/to/video.mp4 bash run_inference.sh

Citation

If you find DreamX-Creator useful in your research, please consider citing our technical report:

@misc{zhu2026dreamxcreatordemocratizingnativeaudiovideo,
  title={DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution},
  author={Jiashu Zhu and Yanhao Zheng and Ruitian Tian and Rujing Dang and Shen Zhang and Bingze Song and Jiachen Lei and Ruimin Lin and Jiahong Wu and Xiangxiang Chu},
  year={2026},
  eprint={2608.31106},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.31106},
}

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

Acknowledgement

We would like to thank the Wan Team, the OpenMOSS Team, and the VideoX-Fun Team for their outstanding open-source work on Wan, MOVA, and VideoX-Fun, respectively.