EmbeddingGemma 2 is an open-source, multimodal embedding model developed by Google DeepMind, designed to map text, images, video, and audio into a unified 768-dimensional latent space. With a total footprint of 740M parameters, the architecture comprises a 270M parameter text backbone (a 130M transformer and 140M embedder) augmented by modular vision (170M) and audio (300M) encoders. This modularity allows developers to selectively load only the necessary encoders, optimizing the model for on-device applications with varying resource constraints.

Core Methodology and Architecture

The model leverages a 24-layer transformer architecture with a 512 model dimension and 2048 hidden dimension. To manage long-range dependencies efficiently, it employs a sliding window of 1024 tokens within a total context window of 8,192 tokens. It utilizes Grouped Query Attention (GQA) and Multi-Query Attention (MQA) with a 5:1 local-to-global attention ratio and a gated FFN with GELU activation. Final representations are derived via mean pooling.

A primary technical innovation is the implementation of Matryoshka Representation Learning (MRL), which allows the model to produce nested, scale-adaptive embeddings. This enables the truncation of the native 768-dimensional vector into lower-dimensional representations (128d, 256d, or 512d) to facilitate reduced storage costs and faster similarity computations. The model requires L2-normalization post-truncation to maintain distance metric integrity:
$$\mathbf{v}_{truncated} = \frac{\mathbf{v}_{1:k}}{\|\mathbf{v}_{1:k}\|_2}$$
where $k \in \{128, 256, 512\}$ and $\mathbf{v}$ is the original 768d vector.

Multimodal Processing and Integration

EmbeddingGemma 2 supports interleaved multimodal inputs, where text and media are combined within a single sequence using specific placeholder tokens: $<|image|>$, $<|video|>$, and $<|audio|>$. These placeholders occupy the shared 8,192-token context window with fixed costs:

• Text: 1 token per subword.
• Images: Default 280 tokens.
• Video: Default 140 tokens per frame (sampled at 1 FPS).
• Audio: 25 tokens per second (16 kHz mono).

Optimization and Task Adaptation

The model utilizes task-steered representations via instruction prefixes. By prepending specific text prefixes (e.g., task: search result | query: {query}), the model optimizes its internal semantic representations for distinct retrieval, clustering, or classification tasks.

Performance and Reliability

The model was trained on a diverse dataset spanning 140+ languages and various codebases, with a data cutoff of January 2025. It achieves state-of-the-art results for its size class, notably reaching 61.36 on the MTEB (multilingual) benchmark and 78.68 on MTEB (code). To ensure deployment stability, the model mandates the use of bfloat16 or float32 precision, as the activation range of EmbeddingGemma 2 exceeds the dynamic range of standard float16, which would otherwise lead to numerical instability or silent representation degradation.