This paper explores the architecture and mechanism behind Multimodal Large Language Models (MLLMs), focusing on how vision-language models bridge the gap between image data and text-based LLMs.

1. Vision Transformer (ViT)

The shift from Convolutional Neural Networks (CNNs) to Vision Transformers (ViT) represents a pivotal change in computer vision. ViT applies the Transformer encoder architecture—which achieved state-of-the-art results in NLP—to image data.

• Tokenization via Patching: Since images lack discrete "words," ViT divides an image into a series of fixed-size patches (typically $16 \times 16$ pixels).
• Linear Projection: Each patch is flattened and linearly embedded into a numerical vector, functioning analogous to word tokens.
• Processing: Once converted into embeddings, these patches are treated as a sequence, enabling the model to utilize standard self-attention mechanisms to understand spatial relationships and global context, effectively allowing the model to "see."

2. BLIP-2: Bridging the Modality Gap

BLIP-2 addresses the computational infeasibility of training massive multimodal models from scratch by introducing a Querying Transformer (Q-Former) to bridge pre-trained vision encoders (like ViT) and pre-trained LLMs.

• Q-Former Architecture: It consists of two sub-modules that share self-attention layers: a vision-side transformer that interacts with the (frozen) vision encoder, and a language-side transformer that interacts with the (frozen) LLM.
• Training Phases:
  • Phase 1 (Representation Learning): The Q-Former is trained using image-caption pairs through three optimization objectives:
    1. Image-Text Contrastive Learning: Aligning image and text embeddings in a shared space.
    2. Image-Text Matching: Binary classification of whether an image-text pair is positive or negative.
    3. Image-Grounded Text Generation: Generating captions based on visual input.
  • Phase 2 (LLM Integration): The learned visual embeddings are transformed via a linear projection layer into "soft visual prompts." These act as a conditional context for the LLM, effectively allowing the text-only model to interpret the visual information extracted by the Q-Former.

3. Applications and Ecosystem

The paper highlights that this modular approach—using a frozen vision encoder and a frozen LLM connected by a trainable bridge—has become the standard for modern MLLMs such as LLaVA and Idefics 2.

The practical implementation involves:

1. Preprocessing: Converting raw images into the specific pixel format required by the vision encoder.
2. Encoding: Extracting visual features through the ViT.
3. Prompting: Projecting those features into the LLM’s input space to generate text, such as descriptive captions, based on the visual context.

By leveraging pre-trained components, this architecture enables the creation of powerful multimodal systems without the extreme computational burden of training from scratch.