Duplicate from gudaochangsheng/RefAlign-1.3B

Co-authored-by: Lei Wang <gudaochangsheng@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+asserts/abstract-refalign.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: mit
+datasets:
+- BestWishYsh/OpenS2V-5M
+- ZhuoweiChen/Phantom-data-Koala36M
+base_model:
+- Wan-AI/Wan2.1-T2V-1.3B
+pipeline_tag: image-text-to-video
+---
+
+
+# 🚀 RefAlign: Representation Alignment for Reference-to-Video Generation
+
+[![arXiv](https://img.shields.io/badge/arXiv-RefAlign-<COLOR>.svg)](https://arxiv.org/abs/2603.25743) [![arXiv](https://img.shields.io/badge/paper-RefAlign-b31b1b.svg)](https://arxiv.org/pdf/2603.25743) ![Visitors](https://visitor-badge.laobi.icu/badge?page_id=gudaochangsheng/RefAlign) [![HF-1.3B](https://img.shields.io/badge/HF-RefAlign--1.3B-yellow?logo=huggingface)](https://huggingface.co/gudaochangsheng/RefAlign-1.3B)
+[![HF-14B](https://img.shields.io/badge/HF-RefAlign--14B-yellow?logo=huggingface)](https://huggingface.co/gudaochangsheng/RefAlign-14B)
+[![MS-1.3B](https://img.shields.io/badge/ModelScope-RefAlign--1.3B-blue)](https://www.modelscope.cn/models/gudaochangsheng98/RefAlign-1.3B)
+[![MS-14B](https://img.shields.io/badge/ModelScope-RefAlign--14B-blue)](https://www.modelscope.cn/models/gudaochangsheng98/RefAlign-14B)
+[![Code](https://img.shields.io/badge/Code-RefAlign-black?style=flat&logo=github)](https://github.com/gudaochangsheng/RefAlign)
+[![Project Page](https://img.shields.io/badge/Project-Page-2ea44f?style=flat-square)](https://gudaochangsheng.github.io/RefAlign-Page/)
+
+<div align="center">
+  <a href="https://gudaochangsheng.github.io/">Lei Wang</a><sup>1,2,*,&ddagger;</sup>,
+  <a href="https://scholar.google.com/citations?hl=zh-TW&user=1uL_9HAAAAAJ">Yuxin Song</a><sup>2,&ddagger;</sup>,
+  <a href="https://github.com/Martinser">Ge Wu</a><sup>1</sup>,
+  <a href="https://scholar.google.com.hk/citations?user=pnuQ5UsAAAAJ&hl=zh-CN&oi=ao">Haocheng Feng</a><sup>2</sup>,
+  <a href="https://hangz-nju-cuhk.github.io/">Hang Zhou</a><sup>2</sup>,
+  <a href="https://jingdongwang2017.github.io/">Jingdong Wang</a><sup>2</sup>
+    <a href="https://yaxingwang.github.io/">Yaxing Wang</a><sup>4&dagger;</sup>
+    <a href="https://scholar.google.com.hk/citations?user=6CIDtZQAAAAJ&hl=en">Jian Yang</a><sup>1,3&dagger;</sup>
+</div>
+
+<div align="center">
+  <sup>1</sup> PCA Lab, VCIP, College of Computer Science, Nankai University &nbsp;&nbsp;
+  <sup>2</sup> Baidu Inc. &nbsp;&nbsp;
+  <sup>3</sup> PCA Lab, School of Intelligence Science and Technology, Nanjing University &nbsp;&nbsp;
+  <sup>4</sup> College of Artificial Intelligence, Jilin University
+</div>
+
+<div align="center">
+  &dagger;Corresponding authors *Interns in Baidu Inc. &ddagger;Equal Contribution
+</div>
+
+<div align="center">
+<img src="asserts/abstract-refalign.png" alt="demo" style="width: 100%;" />
+  <br>
+</div>
+
+---
+
+## 🏆 OpenS2V-Eval Leaderboard
+
+> RefAlign achieves **state-of-the-art performance** on [OpenS2V-Eval](https://huggingface.co/spaces/BestWishYsh/OpenS2V-Eval) across multiple metrics.
+
+| Model | Venue | TotalScore ↑ | Aesthetic ↑ | MotionSmoothness ↑ | MotionAmplitude ↑ | FaceSim ↑ | GmeScore ↑ | NexusScore ↑ | NaturalScore ↑ |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| 🥇 **RefAlign-14B (Ours)** | Open-Source | **60.42%** | 46.84% | **97.61%** | 22.48% | **55.23%** | 68.32% | **48.52%** | 73.63% |
+| 🥇 **RefAlign-1.3B (Ours)** | Open-Source | **56.30%** | 42.96% | 94.74% | 20.74% | 53.06% | 66.85% | 43.97% | 66.25% |
+| Saber | Closed-Source | 57.91% | 42.42% | 96.12% | 21.12% | 49.89% | 67.50% | 47.22% | 72.55% |
+| VINO | Open-Source | 57.85% | 45.92% | 94.73% | 12.30% | 52.00% | 69.69% | 42.67% | 71.99% |
+| BindWeave | Closed-Source | 57.61% | 45.55% | 95.90% | 13.91% | 53.71% | 67.79% | 46.84% | 66.85% |
+| VACE-14B | Open-Source | 57.55% | **47.21%** | 94.97% | 15.02% | 55.09% | 67.27% | 44.08% | 67.04% |
+| Phantom-14B | Open-Source | 56.77% | 46.39% | 96.31% | **33.42%** | 51.46% | **70.65%** | 37.43% | 69.35% |
+| Kling1.6 | Closed-Source | 56.23% | 44.59% | 86.93% | **41.60%** | 40.10% | 66.20% | 45.89% | **74.59%** |
+| Phantom-1.3B | Open-Source | 54.89% | 46.67% | 93.30% | 14.29% | 48.56% | 69.43% | 42.48% | 62.50% |
+| MAGREF-480P | Open-Source | 52.51% | 45.02% | 93.17% | 21.81% | 30.83% | 70.47% | 43.04% | 66.90% |
+| SkyReels-A2-P14B | Open-Source | 52.25% | 39.41% | 87.93% | 25.60% | 45.95% | 64.54% | 43.75% | 60.32% |
+| Vidu2.0 | Closed-Source | 51.95% | 41.48% | 90.45% | 13.52% | 35.11% | 67.57% | 43.37% | 65.88% |
+
+## 📦 Model Weights
+
+| Model | Params | Hugging Face | ModelScope |
+|---|---:|---|---|
+| RefAlign-1.3B | 1.3B | [![HF Download](https://img.shields.io/badge/HuggingFace-Download-yellow?logo=huggingface)](https://huggingface.co/gudaochangsheng/RefAlign-1.3B) | [![MS Download](https://img.shields.io/badge/ModelScope-Download-blue)](https://www.modelscope.cn/models/gudaochangsheng98/RefAlign-1.3B) |
+| RefAlign-14B | 14B | [![HF Download](https://img.shields.io/badge/HuggingFace-Download-yellow?logo=huggingface)](https://huggingface.co/gudaochangsheng/RefAlign-14B) | [![MS Download](https://img.shields.io/badge/ModelScope-Download-blue)](https://www.modelscope.cn/models/gudaochangsheng98/RefAlign-14B) |
+
+> ⚠️ **Note**
+>
+> The provided weights are **DiT (Diffusion Transformer) checkpoints fine-tuned from Wan2.1**.  
+> To run RefAlign, please:
+>
+> 1. Download the original **[Wan2.1](https://huggingface.co/collections/Wan-AI/wan21)** model (including VAE, text encoder, etc.).
+> 2. Replace the **DiT weights** in Wan2.1 with the RefAlign weights provided above.
+>
+> No modification is required for other components.
+
+## 🎬 Inference
+
+
+```shell
+# Inference RefAlign-1.3B
+python examples/wanvideo/model_inference/Wan2.1-T2V-1.3B_subject.py
+
+# Inference RefAlign-14B
+python examples/wanvideo/model_inference/Wan2.1-T2V-14B_subject.py
+  ```
+## Citation
+
+If you find RefAlign useful, please consider giving our repository a star (⭐) and citing our [paper](https://arxiv.org/abs/2603.25743).
+
+```
+@misc{wang2026refalign,
+  title={RefAlign: Representation Alignment for Reference-to-Video Generation},
+  author={Lei Wang and Yuxin Song and Ge Wu and Haocheng Feng and Hang Zhou and Jingdong Wang and Yaxing Wang and Jian Yang},
+  year={2026},
+  eprint={2603.25743},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV}
+}
+```
+## Acknowledgement
+
+This project is based on [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). Thanks for their awesome works.
+We sincerely acknowledge the excellent and inspiring prior work, [Phantom](https://github.com/Phantom-video/Phantom), [VINO](https://sotamak1r.github.io/VINO-web/), [OpenS2V](https://github.com/PKU-YuanGroup/OpenS2V-Nexus), [Phantom-Data](https://phantom-video.github.io/Phantom-Data/) and [Wan2.1](https://wan.video/).
+## Contact
+If you have any questions, please feel free to reach out to me at  `scitop1998@gmail.com`. 

config.json

@@ -0,0 +1,7 @@
+{
+  "model_type": "refalign",
+  "architectures": ["RefAlignForConditionalGeneration"],
+  "hidden_size": 0,
+  "num_hidden_layers": 0,
+  "note": "This config file is added to enable Hugging Face Hub download statistics tracking."
+}

diffusion_pytorch_model-refalign.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc3698513712e22693c0d7fb2669298ba19af55167658308c9ad413a556df474
+size 2838077304