Update documentation to Please add how to character swap for Ref2V/Image to Video.
I see some videos online now where people are successfully replacing people in a video with their image references but I have failed to do this no matter how many times and specific I word the prompt.
It will not transfer the subject in image 1 to replace the person in the reference video especially not preserving them across cuts.
It will also not transfer the subject in image 1 to replace the person in image 2 and image 3 if using only images.
Minimax seems to always cut to image 2 and 3 in the shots using the people in those images instead and ignore me asking it to put the subject of image 1 into those shots replacing the people in the other images.
I don't understand why.
I have followed the document but it isn't working https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
My current prompt has the person from image 1 only working for shot 1, shot 2 and 3 will not make the person in image 1 override the people in image 2 and 3.
#################
<Subject 1> is the woman whose appearance comes from <Picture 1> and whose spoken words will come from <Audio 1>.
<Picture 2> and <Picture 3> is a storyboard reference for [Shot 2] and [Shot 3], defining their viewpoint, subject placement, and shot order.
<Subject 1> (appears in [Shot 1], [Shot 2], [Shot 3]): fully_preserved.
[0.00s: Shot 1] Camera fade to black showing <Subject 1> woman's face as she walks into the room.
6.00 [Shot 2]: The camera cuts to her lying on her back sitting on the couch with the camera close up to her face as she says the words directly from <Audio 1> [English] Hey hows it going....Oh you are watching a movie?.
10.00 [Shot 3]: The camera cuts to her from a new close up angle as she says the words directly from <Audio 1> [English] I really hate this movie you know!.
#################
Result: Uses shot 1 subject 1 from the reference image correctly but for Shot 2 and 3 it relies on the image 2 and 3 perfectly and uses the same camera angle and closeups of those images but it also uses the person in those images and is not replacing that person with the subject 1 from image 1!
How to do this correctly thank you?
Same for me
I think the important distinction here is between:
- an image that defines a subject identity, and
- an image that acts as a standalone keyframe/storyboard reference.
The current reference prompt guide treats those differently.
If Picture 1 defines the person you want to preserve, I would make that identity relationship explicit:
<Subject 1> is the woman whose appearance comes from <Picture 1>.
Her identity, facial features, hairstyle and clothing identity remain consistent throughout all shots.
For Picture 2 and Picture 3, if you only want their composition / camera placement, I would avoid wording that makes the people inside those images look like subjects that should also be preserved.
Instead describe them narrowly as planning references:
<Picture 2> is a storyboard reference for Shot 2.
Only its camera angle, framing and subject placement are referenced.
The identity and appearance of the person visible in Picture 2 are not referenced.
<Picture 3> is a storyboard reference for Shot 3.
Only its camera angle, framing and composition are referenced.
The identity and appearance of the person visible in Picture 3 are not referenced.
Then make Subject 1 explicit in every shot:
[Shot 1] <Subject 1> ...
[Shot 2] <Subject 1> appears in the framing and camera arrangement referenced from <Picture 2> ...
[Shot 3] <Subject 1> appears in the framing and camera arrangement referenced from <Picture 3> ...
The official reference guide specifically says that when an image is used only to define a character, it should be cited inside the <Subject N> definition; when an image is used as storyboard/shot-planning reference, the prompt should state which shots it maps to and what planning information it provides.
So I would try to reduce the semantic role of Picture 2/3 from:
“reference images containing people”
to:
“camera/composition-only storyboard references.”
I would also avoid repeatedly describing the appearance of the people inside Picture 2/3.
That won't guarantee the model ignores their identities completely, but it gives the conditioning structure a much clearer separation between:
who the subject is
and
how the shot should be composed.