View original
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Paper

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Anton Obukhov
2026.09.09
·Arxiv·by Homin.Lee
#Computer Vision#Deep Learning#Diffusion Transformers#Image Editing#Monocular Depth Estimation

Key Points

  • 1Marigold V2 introduces a cost-effective fine-tuning protocol that repurposes an open-source image-editing diffusion transformer into a high-performance monocular depth estimator using only a single consumer GPU.
  • 2The method improves geometric accuracy and structural fidelity by implementing iREPA-depth, a novel feature alignment loss that uses ground-truth depth to supervise internal model representations.
  • 3Through the integration of SinkLoss—a Sinkhorn-based objective—the model effectively minimizes flying pixel artifacts and preserves intricate edge details, outperforming existing state-of-the-art approaches across major benchmarks.

Marigold V2 is a monocular depth estimation framework that repurposes the open-source image-editing diffusion transformer (DiT), Qwen-Image-Edit, into a state-of-the-art dense regression model. By focusing on a cost-effective, two-stage fine-tuning protocol using 4-bit QLoRA, the authors address the common limitations of diffusion-based depth models, specifically the loss of fine-grained structural detail and the presence of over-smoothed boundaries.

Core Methodology

The model operates by transforming RGB latents into normalized log-depth latents in a single-step inference pass, leveraging the generative priors inherent in the DiT. The training process consists of two primary stages:

Stage 1: DiT Adaptation and Semantic Alignment
The authors initialize with Qwen-Image-Edit and apply 4-bit quantization and rank-128 QLoRA adapters. The training objective incorporates latent-space rectified-flow supervision, pixel-space reconstruction, and a novel semantic alignment technique called iREPA-depth. Unlike prior methods that align features against RGB inputs, iREPA-depth aligns the DiT's internal representations with semantic features extracted from ground-truth depth maps using a frozen DINOv3 encoder. This provides a more direct geometric supervisory signal, facilitating faster convergence and improved local structural fidelity. The Stage 1 objective is defined as:
LDiT=λlatentLlatent+λpixLpix+λgradLgrad+λiREPALiREPAL_{DiT} = \lambda_{latent}L_{latent} + \lambda_{pix}L_{pix} + \lambda_{grad}L_{grad} + \lambda_{iREPA}L_{iREPA}

Stage 2: SinkLoss Refinement
To handle the inherent noise and ambiguity in ground-truth datasets—which often contain stochastic depth sampling for thin or transparent objects—the authors introduce SinkLoss. Instead of strictly enforcing pixel-wise alignment, the image is partitioned into K×KK \times K blocks. Within these blocks, the model uses Sinkhorn–Knopp matching to establish a soft one-to-one assignment between predicted and ground-truth depth values. The cost matrix is defined as Cij=∣d^i−dj∣C_{ij} = |\hat{d}_i - d_j|, and the Sinkhorn iteration solves for a transport plan MM that minimizes transport cost while maximizing entropy:
M=arg⁡min⁡M∈U⟨M,C~⟩−τH(M)M = \arg \min_{M \in U} \langle M, \tilde{C} \rangle - \tau H(M)
This mechanism allows the model to ignore precise pixel-wise noise in the ground truth while maintaining consistent depth distributions, resulting in sharper edges and reduced "flying pixel" artifacts.

Key Outcomes

Marigold V2 achieves state-of-the-art zero-shot performance across standard benchmarks (NYUv2, KITTI, ETH3D, ScanNet, and DIODE), reporting significant improvements in AbsRel and δ1\delta_1 accuracy. Beyond depth, the recipe proves to be a versatile foundation for other dense regression tasks, including surface normal estimation, intrinsic image decomposition, and depth completion. The final model retains the efficiency of a single-pass VAE-based architecture, making it highly effective for both computational photography and real-time graphics applications.