| | --- |
| | language: |
| | - "en" |
| | tags: |
| | - video |
| | license: apache-2.0 |
| | pipeline_tag: text-to-video |
| | library_name: diffusers |
| | --- |
| | |
| | <p align="center"> |
| | <img src="assets/logo.jpg" height=30> |
| | </p> |
| |
|
| | # FastMochi Model Card |
| |
|
| | ## Model Details |
| |
|
| | <div align="center"> |
| | <table style="margin-left: auto; margin-right: auto; border: none;"> |
| | <tr> |
| | <td> |
| | <img src="assets/mochi-demo.gif" width="640" alt="Mochi Demo"> |
| | </td> |
| | </tr> |
| | <tr> |
| | <td style="text-align:center;"> |
| | Get 8X diffusion boost for Mochi with FastVideo |
| | </td> |
| | </tr> |
| | </table> |
| | </div> |
| | |
| | FastMochi is an accelerated [Mochi](https://huggingface.co/genmo/mochi-1-preview) model. It can sample high quality videos with 8 diffusion steps. That brings around 8X speed up compared to the original Mochu with 64 steps. |
| |
|
| | - **Developed by**: [Hao AI Lab](https://hao-ai-lab.github.io/) |
| | - **License**: Apache-2.0 |
| | - **Distilled from**: [Mochi](https://huggingface.co/genmo/mochi-1-preview) |
| | - **Github Repository**: https://github.com/hao-ai-lab/FastVideo |
| |
|
| | ## Usage |
| |
|
| | - Clone [Fastvideo](https://github.com/hao-ai-lab/FastVideo) repository and follow the inference instructions in the README. |
| | - You can also run FastMochi using the official [Mochi repository](https://github.com/Tencent/HunyuanVideo) with the script below and this [compatible weight](https://huggingface.co/FastVideo/FastMochi). |
| |
|
| | <details> |
| | <summary>Code</summary> |
| |
|
| | ```python |
| | from genmo.mochi_preview.pipelines import ( |
| | DecoderModelFactory, |
| | DitModelFactory, |
| | MochiMultiGPUPipeline, |
| | T5ModelFactory, |
| | linear_quadratic_schedule, |
| | ) |
| | from genmo.lib.utils import save_video |
| | import os |
| | |
| | with open("prompt.txt", "r") as f: |
| | prompts = [line.rstrip() for line in f] |
| | |
| | pipeline = MochiMultiGPUPipeline( |
| | text_encoder_factory=T5ModelFactory(), |
| | world_size=4, |
| | dit_factory=DitModelFactory( |
| | model_path=f"weights/dit.safetensors", model_dtype="bf16" |
| | ), |
| | decoder_factory=DecoderModelFactory( |
| | model_path=f"weights/decoder.safetensors", |
| | ), |
| | ) |
| | # read prompt line by line from prompt.txt |
| | |
| | |
| | output_dir = "outputs" |
| | os.makedirs(output_dir, exist_ok=True) |
| | for i, prompt in enumerate(prompts): |
| | video = pipeline( |
| | height=480, |
| | width=848, |
| | num_frames=163, |
| | num_inference_steps=8, |
| | sigma_schedule=linear_quadratic_schedule(8, 0.1, 6), |
| | cfg_schedule=[1.5] * 8, |
| | batch_cfg=False, |
| | prompt=prompt, |
| | negative_prompt="", |
| | seed=12345, |
| | )[0] |
| | save_video(video, f"{output_dir}/output_{i}.mp4") |
| | ``` |
| |
|
| | </details> |
| |
|
| |
|
| | ## Training details |
| |
|
| | FastMochi is consistency distillated on the [MixKit](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main) dataset with the following hyperparamters: |
| | - Batch size: 32 |
| | - Resulotion: 480X848 |
| | - Num of frames: 169 |
| | - Train steps: 128 |
| | - GPUs: 16 |
| | - LR: 1e-6 |
| | - Loss: huber |
| |
|
| | ## Evaluation |
| | We provide some qualitative comparisons between FastMochi 8 step inference v.s. the original Mochi with 8 step inference: |
| |
|
| |
|
| | | FastMochi 6 steps | Mochi 6 steps | |
| | | --- | --- | |
| | |  |  | |
| | |  |  | |
| | |  |  | |
| | |  |  | |
| |
|
| |
|
