Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation (🎉 CVPR 2026 Highlight 🎉)

Zihao Wang, Yuxiang Wei, Xinpeng Zhou, Tianyu Zhang, Tao Liang, Yalong Bai, Hongzhi Zhang, Wangmeng Zuo

Harbin Institute of Technology, Duxiaoman

Model Description

Premier is a personalized text-to-image generation framework that modulates user preferences through learnable embeddings. It leverages FLUX.1-dev as the base model and introduces:

• Learnable User Embedding: Individual trainable vectors for each user, optimized via Flow Matching Loss and Dispersion Loss.
• Prompt Preference Modulation (PPM): Context-aware modulation for fine-grained, semantically aligned preference injection.
• Lightweight: Single-user embedding ≈61 KB, fast training (≈30 min), minimal inference overhead (≈1 second).

🔗 Project Page | 📄 Paper | 💻 Code

user_embedding.safetensors contains embeddings for 1000 training users (BF16, shape [1000, 30720]). The users/ and users_linear/ directories contain individually fine-tuned weights for 50 test users (IDs 3685~4279), which are distinct from the training set.

Model Files

File	Description	Size
mod_adapter.safetensors	Modulation adapter weights (trained at 260k steps)	≈1.81 GB
user_embedding.safetensors	Shared user preference embedding (1000 training users)	≈61 MB
adapter_config.yaml	Adapter architecture configuration	-
users/user_embedding_*.safetensors	Per-user embedding weights — 50 test users (non-linear)	≈60 KB each
users_linear/user_combination_*.safetensors	Linear combination user weights — 50 test users (linear)	≈2 KB each

Requirements

• Python 3.12+
• PyTorch 2.6.0
• diffusers 0.33.0
• transformers 4.52.4

Usage

1. Clone the Premier repository and install dependencies

git clone https://github.com/120L020904/Premier.git
cd Premier
pip install -r requirements.txt

2. Download model weights

# Download from HuggingFace (assume saved to ./Premier/)
from huggingface_hub import snapshot_download
snapshot_download("pino10010/Premier", local_dir="./Premier")

After download, the directory structure should be:

./Premier/
├── adapter_config.yaml
├── mod_adapter.safetensors
├── user_embedding.safetensors
├── users/
│   └── user_embedding_*.safetensors
└── users_linear/
    └── user_combination_*.safetensors

3. Inference with shared user embedding (training set users)

import os
import sys
import torch
from diffusers import FluxPipeline
from safetensors.torch import load_file
from torch import nn

sys.path.append("path/to/Premier")
from scripts.pipeline.flux_adapter import generate_xverse
from scripts.pipeline.mod_adapters import load_modulation_adapter
from scripts.utils.utils import get_config, save_images

device = "cuda"
dtype = torch.bfloat16
model_dir = "./Premier"

# Load FLUX.1-dev base model
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=dtype
).to(device)

# Load adapter config and weights
adapter_config = get_config(config_path=os.path.join(model_dir, "adapter_config.yaml"))
mod_adapter = load_modulation_adapter(
    adapter_config, dtype, device,
    ckpt_dir=model_dir,
    is_training=False
)
mod_adapter.eval()

# Load shared user embedding (1000 training users, BF16, shape [1000, 30720])
user_token_num = adapter_config["model"]["modulation"]["user_token_num"]  # 30
state_dict = load_file(os.path.join(model_dir, "user_embedding.safetensors"))
user_embedding = nn.Embedding(
    num_embeddings=1000,
    embedding_dim=user_token_num * 1024
).to(device=device, dtype=dtype)
user_embedding.load_state_dict(state_dict)

# Generate image for a training-set user (IDs 0~999)
user_id = 0
indices = torch.tensor([user_id], dtype=torch.long).to(device)
user_pref = user_embedding(indices).view(-1, user_token_num, 1024)

prompt = "a cute cat sitting on a windowsill in watercolor style"
generator = torch.Generator(device).manual_seed(42)

result = generate_xverse(
    pipeline=pipe,
    mod_adapter=mod_adapter,
    user_preference_embedding=user_pref,
    prompt=prompt,
    prompt_2=prompt,
    num_inference_steps=30,
    guidance_scale=2.5,
    height=512,
    width=512,
    generator=generator,
    model_config=adapter_config,
)
image = result.images[0]
image.save("output.png")

4. Inference with per-user embedding (non-linear)

# Load individual user embedding (non-linear, for user 3685)
user_id = 3685
user_weights = load_file(os.path.join(model_dir, f"users/user_embedding_{user_id}.safetensors"))
user_token_num = 30

train_user_embedding = nn.Embedding(
    num_embeddings=1,
    embedding_dim=user_token_num * 1024
).to(device=device, dtype=dtype)
train_user_embedding.load_state_dict(user_weights)

indices = torch.tensor([0], dtype=torch.long).to(device)
user_pref = train_user_embedding(indices).view(-1, user_token_num, 1024)

# Use same generate_xverse() call as above with user_pref

5. Inference with per-user linear combination

from scripts.train_flux.train_user_embedding_linear import EmbeddingLinearCombination

user_id = 3685
user_token_num = 30

# Load shared training embedding
train_state_dict = load_file(os.path.join(model_dir, "user_embedding.safetensors"))
train_user_embedding = nn.Embedding(
    num_embeddings=1000,
    embedding_dim=user_token_num * 1024
).to(device=device, dtype=dtype)
train_user_embedding.load_state_dict(train_state_dict)

# Load linear combination weights
combination_state_dict = load_file(
    os.path.join(model_dir, f"users_linear/user_combination_{user_id}.safetensors")
)
embedding_comb = EmbeddingLinearCombination(
    combination_size=1, embedding_num=1000, use_softmax=False
).to(device=device, dtype=dtype)
embedding_comb.load_state_dict(combination_state_dict)

indices = torch.tensor([0], dtype=torch.long).to(device)
user_pref = embedding_comb(
    train_user_embedding, input_ids=indices
).view(-1, user_token_num, 1024)

# Use same generate_xverse() call as above with user_pref

Citation

@article{wang2026premier,
  title={Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation},
  author={Wang, Zihao and Wei, Yuxiang and Zhou, Xinpeng and Zhang, Tianyu and Liang, Tao and Bai, Yalong and Zhang, Hongzhi and Zuo, Wangmeng},
  journal={arXiv preprint arXiv:2603.20725},
  year={2026}
}

Acknowledgment

Thanks to OminiControl, XVerse, and PrefGen.