Could you share an example workflow?
Hi, thank you for training this model. I wish to use it but I am having a hard time making an workflow for this. Seems like the model needs custom version of ComfyUI-Advanced-ControlNet and I installed it. However I couldn't get it to work as it fails to load the ControlNet model.
Could you share an example workflow? That would be a great help.
Are you using the Load ControlNet node from Comfy Core or from the custom node package? Comfy Core's Load ControlNet does not support this model. I can share a workflow later, but I am not at my computer right now.
I used "Load Advanced ControlNet Model" from the custom node package.
Q1. Between the two, which one should I use?
The problem was that I pulled the git repository but it was on main branch, not feat/anima-vace-controlnet. I moved the HEAD to the branch and fixed the problem. However I encountered another problem which doesn't generate images guided by the ControlNet. The generated image does have contour of the reference image but is full of noises.
Q2. Am I supposed to "Apply ControlNet" node from custom node package as well? I had been using Comfy Core one. I'll try using it and get back to you.
I tried using the above instead of the below but didn't fix the issue. Hmm I am lost. Seeing the actual example workflow might be of a great help. I'll be waiting. Take your time.
I somehow managed to make it work but I have encountered two weird issues.
AssertionError: H,W (180, 135) should be divisible by spatial_patch_size 2
When I try to generate an image of1080x1440, I get this error. Only way to resolve it was restarting ComfyUI and having empty latent image to be1088x1440. Does the your ControlNet model require an image to be in a certain size? I never encountered such error messages when I was messing around with SDXL/Illustrious ControlNet models. Specifically, Xinsir's Union ControlNet model.Image quality degrades to be like blurry and noisy after encountering the first issue. When I deliberately make errors with
1080x1440sized empty latent images, the generated images become a chunk of noises rather an image.
Here's my workflow
Reference Image
Before Error
After First Error
After Second Error
After Third Error
Issue 1: I have encountered this issue but I did not push the fix up. Essentially what happens is the vae compresses the image by 8 (1080->135), and then Anima further patchifies the image by 2 (135->?), and thus every 16*16 blocks becomes a single token. 1080 is not divisible by 16, and thus you see the crash. The main Anima branch forward resolve this simply by padding the input (135->136), and then cropping the output back (which is standard practice in other models too), but my fork does not do this right now. I will push the fix up after some cleanup.
I am not sure I quite understand the second issue. Do you mean after the crash at 1080 resolution once, you adjusted the resolution to 1088 to be able to generate, but the image becomes blurry? It might be because the crash caused the model to not unpatch correctly, so when you generate a second time, the model is patched again, essentially applying the CN twice, three times, etc. I haven't verified this hypothesis, but the issue should go away anyway assuming the crash is fixed.
Thank you. By the way, do you have plans to make PRs to https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet? Seems like the original repository also added support for kohya-ss's LLLite ControlNets. It may be great if relevant features could be accessed from single extension.
Regarding the second issue, yes your understanding is correct. It gets blurrier every time there are crashes at 1080 resolution. Like you mentioned, it should be irrelevant once you push the fix.
I have made a commit containing the fix.
With regards to merging into the original ACN: I have opened an issue over there and the author replied he will be supporting this.
Awesome. Thank you very much!
I tried it but couldn't generate an image. Can you share a working ComfyUI workflow?
I tried it but couldn't generate an image. Can you share a working ComfyUI workflow?
The attached workflow in this thread should be working. What does not work on your end/what is your workflow?
Hello! Not sure what i'm doing wrong, i'm using custom nodes to load the model, the same goes for control net, i even tried to use workflow from this thread, result is the same ;
ComfyUI Error Report
Error Details
- Node ID: 123
- Node Type: ACN_ControlNetLoaderAdvanced
- Exception Type: Exception
- Exception Message: Exception: Something went wrong when loading 'i:\ComfyImages\models\controlnet\anima-vace-depth.safetensors'; ControlNet is None.
ok, i have read more carefull and CrownBerry advice about git repository and main branch solved my issue, everything working now. Big thanks TaihoC for the model and CrownBerry for helpfull comments.
EDIT: Just had to update ComfyUI. In case anyone else faces same issue, just go to ComfyUI manager and click on Update All, then restart.
I'm still getting an error after switching to the other branch.
ComfyUI Error Report
Error Details
- Node ID: 78
- Node Type: KSampler
- Exception Type: RuntimeError
- Exception Message: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16
Stack Trace
File "/workspace/ComfyUI/execution.py", line 524, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/execution.py", line 333, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/execution.py", line 307, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/workspace/ComfyUI/execution.py", line 295, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/workspace/ComfyUI/nodes.py", line 1593, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/nodes.py", line 1558, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/sample.py", line 66, in sample
samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 1177, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 1067, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 1049, in sample
output = executor.execute(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 993, in outer_sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed, latent_shapes=latent_shapes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 979, in inner_sample
samples = executor.execute(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/custom_nodes/ComfyUI-vectorscope-cc/callback_manager.py", line 55, in sample
return original_sample(
^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 751, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/k_diffusion/sampling.py", line 218, in sample_euler_ancestral
return sample_euler_ancestral_RF(model, x, sigmas, extra_args, callback, disable, eta, s_noise, noise_sampler)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/k_diffusion/sampling.py", line 247, in sample_euler_ancestral_RF
denoised = model(x, sigmas[i] * s_in, **extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 400, in __call__
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 952, in __call__
return self.outer_predict_noise(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 959, in outer_predict_noise
).execute(x, timestep, model_options, seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 962, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 380, in sampling_function
out = calc_cond_batch(model, conds, x, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 205, in calc_cond_batch
return _calc_cond_batch_outer(model, conds, x_in, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 213, in _calc_cond_batch_outer
return executor.execute(model, conds, x_in, timestep, model_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/patcher_extension.py", line 112, in execute
return self.original(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/samplers.py", line 320, in _calc_cond_batch
c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond), transformer_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet-Anima/adv_control/utils.py", line 700, in get_control_inject
return self.get_control_advanced(x_noisy, t, cond, batched_number, transformer_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet-Anima/adv_control/control_anima_vace.py", line 473, in get_control_advanced
x_B_T_H_W_D, rope_emb, extra_pos_emb = dit.prepare_embedded_sequence(x_5d)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/ldm/cosmos/predict2.py", line 795, in prepare_embedded_sequence
x_B_T_H_W_D = self.x_embedder(x_B_C_T_H_W)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/ldm/cosmos/predict2.py", line 318, in forward
x = self.proj(x)
^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/container.py", line 250, in forward
input = module(input)
^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ComfyUI/comfy/ops.py", line 362, in forward
return super().forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 134, in forward
return F.linear(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^