MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model

MuCo introduces a novel framework for training multimodal embedding models by addressing the limitations of traditional single-turn contrastive learning (e.g., CLIP-style training), which often fails to capture complex, multi-step semantic relationships and nuanced conversational contexts. The paper proposes a shift toward "Multi-turn Contrastive Learning," where the alignment objective is extended to capture dialogue-like sequences of image-text interactions.

Core Methodology

The fundamental goal of MuCo is to align multimodal embeddings not just in static image-text pairs, but across coherent multi-turn sequences. The methodology is built upon three technical pillars:

1. Multi-turn Data Construction (M3T Dataset)
The authors introduce the M3T (Multi-turn Multimodal Training) dataset, which frames multimodal interaction as a sequential task. Unlike standard contrastive datasets that treat each sample as an independent static pairing, M3T constructs meta-data comprising query-response sequences. This allows the model to learn conditional relationships where a subsequent text query depends on the context of the previous turn and the associated image state.

2. Multi-turn Contrastive Objective
The training framework moves beyond the standard InfoNCE loss applied to singular pairs. Let $(I, \{T_1, T_2, ..., T_n\})$ represent an image and a sequence of $n$ related text turns. The MuCo objective optimizes a contrastive loss $L$ defined as:
$$L = -\sum_{i=1}^{n} \log \frac{\exp(\text{sim}(f(I, T_{1:i-1}), g(T_i)) / \tau)}{\sum_{j=1}^{N} \exp(\text{sim}(f(I, T_{1:j-1}), g(T_j)) / \tau)}$$
where $f$ is the multimodal encoder (conditioning on image $I$ and history $T_{1:i-1}$), $g$ is the text encoder, and $\tau$ is the temperature parameter. This forces the model to encode the query $T_i$ specifically relative to the accumulated multimodal context, improving the representation of intent-driven retrieval.

3. MMEB (Multimodal Embedding Benchmark) Integration
To evaluate the effectiveness of this multi-turn learning, the authors utilize the MMEB framework, providing comprehensive meta-files for sub-benchmarks like CIRR, A-OKVQA, and WebQA. This allows for fine-grained evaluation on tasks requiring reasoning over images. The inclusion of generated captions in the training data acts as a regularization mechanism, ensuring that the model maintains robust semantic grounding even when dialogue history is noisy or incomplete.

Technical Contributions

• Sequential Semantic Alignment: By utilizing the multi-turn structure, MuCo mitigates the "alignment drift" often seen in long-form vision-language tasks, where simple embedding models lose focus on specific visual regions during multi-step reasoning.
• Dataset Efficiency: The release of the M3T dataset (10.8 GB of metadata) provides a standardized, large-scale resource for training agents that require long-context visual understanding.
• Robustness via Retries: The methodology emphasizes practical implementation stability, employing exponential backoff strategies for data sourcing from original URLs to maintain high-fidelity training pairs for the contrastive objective.

In summary, MuCo elevates multimodal embedding training from static, single-point matching to a sequential, context-aware objective, effectively bridging the gap between standard contrastive learning and multi-turn visual-language conversational agents.