arxiv:2609.08084

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Published on Sep 8

· Submitted by

Anton Obukhov on Sep 9

HUAWEI Bayer Lab

Upvote

Authors:

Thiemo Wandel ,

Anton Obukhov ,

Abstract

Marigold V2 repurposes diffusion transformers for monocular depth estimation via single-step flow-matching inference, semantic alignment, and a Sinkhorn-based two-stage fine-tuning protocol, yielding sharper out-of-distribution depth maps and strong results on related dense regression tasks.

Generated by thinkingmachines/Inkling-Small

Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

View arXiv page View PDF Project page GitHub 79 Add to collection

Community

toshas

Paper author Paper submitter about 11 hours ago

Marigold V2 is out! (to appear at SIGGRAPH Asia 2026).

If you missed V1: Marigold post-trains an image generator into a depth estimator on one GPU -- the accessible research game. V2 upgrades to a diffusion transformer: single step, very sharp edges, and quite versatile.

Everything is public today:

Website (paper): https://hf.co/spaces/huawei-bayerlab/marigold-v2-web
Code: https://github.com/huawei-bayerlab/marigold-v2
Weights: https://hf.co/huawei-bayerlab/marigold-v2-0
Demo: https://hf.co/spaces/toshas/Marigold-V2

The video walks through the paper in 90 seconds, here is the full story:

It is made for small labs. Take a big pretrained DiT (Qwen-Image-Edit-2509), quantize it to 4 bit, add a rank-128 QLoRA, and fine-tune on one 32 GB consumer GPU. Sensible depth after a few hours, done in a few days. No 80 GB card, no stack of 8. Inference is a single step and does not OOM at 2K.
We like unconventional choices that stick. Marigold V1 pushed depth maps through an image VAE, which had no business working, and it did. V2 pushes the ground-truth depth through DINOv3 and aligns the DiT features to it (iREPA-depth).
Even synthetic ground truth is not perfect. Hypersim is our cleanest training data, and zoomed in on grass blades its depth is broken: some pixels sit on the blade, others fall through to the background.
Noisy GT needs a loss that tolerates it, so: SinkLoss, no pixel-to-pixel matching. Tile the image into 5x5 blocks and Sinkhorn-match the 25 predicted depths to the 25 ground-truth ones. The match is soft, and the loss is the transport cost.
The recipe covers all the Marigold modalities: depth, surface normals, albedo, just better. Depth beats every model trained on comparable data on NYUv2, KITTI, ETH3D, ScanNet, DIODE. Watch the fur and thin details when the slider goes RGB, V1.1, V2.
Depth for robotics stops at the glass, but in computational photography you need a see-through depth. Fine-tuned on the last ray hit of LayeredDepth-Syn, the same model sees through the window.
And as a bonus, the same model does metric depth completion: give it sparse lidar points, it fits a small LoRA at test time and fills in the rest, no retraining (that last clip is from Marigold-DC).

Thanks to the team: Igor Pavlovic and Thiemo Wandel (equal contribution), Luca Bartolomei, Andrey Davydov, Fabio Tosi, Matteo Poggi, Sabine Süsstrunk, Dengxin Dai. HUAWEI Bayer Lab, EPFL, University of Bologna.