Multimodal Models
On vision-language models, the CLIP revolution, GPT-4V, Gemini, Gemma 4, and the ongoing project of building AI systems that can see, read, and reason across modalities simultaneously.
Multimodal Models
Humans don’t experience the world in text. We see, hear, touch, and then sometimes describe what we perceived in words. Language is a compression of experience, not the experience itself. And yet, for the first seven years of the deep learning revolution, AI’s most impressive systems were either vision models that couldn’t speak or language models that couldn’t see. ResNet could classify images into a thousand categories but couldn’t tell you what was happening in the scene. GPT-3 could write essays about paintings it had never seen. They were savants in isolated sensory channels.
Multimodal models — systems that process and reason across multiple input types, typically vision and language — represent the field’s attempt to undo this artificial separation. The trajectory from CLIP in 2021 to GPT-4V in 2023 to Gemma 4 in 2026 is one of the most important arcs in recent AI history, and it’s still far from complete.1
CLIP and The Contrastive Foundation
The modern multimodal era begins with CLIP (Contrastive Language-Image Pre-training), which I’ve written about in detail. Radford et al. (2021) trained dual encoders — one for images, one for text — on 400 million image-text pairs from the internet, using a contrastive loss that pushed matching pairs together and non-matching pairs apart in a shared embedding space.
CLIP’s contribution was not the idea of multimodal embeddings — that existed before — but the scale and the demonstration that the resulting representations were genuinely useful. Zero-shot CLIP matched supervised ResNet-50 on ImageNet classification. More importantly, CLIP embeddings became the backbone of the entire text-to-image generation ecosystem: DALL-E 2, Stable Diffusion, and hundreds of derivative systems used CLIP (or OpenCLIP) as their text conditioning mechanism.
But CLIP had a fundamental limitation: it could match images with text but couldn’t generate text about images. It produced embeddings, not language. You could use it to ask “is this image more like ‘a dog’ or ‘a cat’?” but not “describe what’s happening in this image.” For that, you needed a different architecture.
The Visual Language Model Paradigm
The next step was connecting vision encoders to language models. The basic recipe, which emerged independently at several labs in 2022-2023:2
Take a pretrained vision encoder (typically CLIP’s image encoder or a Vision Transformer variant).
Add a projection layer (linear, MLP, or cross-attention) that maps visual tokens from the vision encoder into the language model’s embedding space.
Fine-tune on image-text instruction data: images paired with questions and detailed answers.
LLaVA (Liu et al., 2023) was the paper that crystallized this approach. Starting with a CLIP ViT-L/14 vision encoder and a Vicuna 13B language model, connected by a simple linear projection, LLaVA demonstrated that you could build a capable visual question-answering system with minimal architectural innovation. The key insight: the vision encoder and the language model each brought strong pretrained capabilities; you just needed to teach them to talk to each other.3
Flamingo (Alayrac et al., 2022) from DeepMind took a different approach: instead of projecting visual tokens into the language model’s input space, it used gated cross-attention layers inserted between the language model’s transformer blocks. This allowed the model to attend to visual features at every layer of processing, not just the input. Flamingo was not open-sourced, but its architecture influenced many subsequent models including the Idefics family.
The performance gains were rapid. By mid-2023, visual language models could describe images in detail, answer questions about visual content, OCR text from photographs, explain memes, interpret charts and graphs, and solve visual reasoning problems that required understanding spatial relationships, counting, and commonsense inference.
GPT-4V and the Frontier Push
GPT-4V, released in September 2023, was the model that convinced the mainstream that multimodal AI was real and useful. Not because it introduced new techniques — OpenAI’s exact architecture remains unpublished — but because it just worked across an extraordinary range of visual tasks. Upload a whiteboard photo and get a summary. Upload a meal and get a nutritional estimate. Upload a circuit diagram and get an explanation. Upload a chart and ask questions about trends. Upload a math problem and get a worked solution.4
GPT-4V demonstrated something the research community already knew but hadn’t demonstrated at consumer scale: multimodal understanding is more than the sum of vision + language. When you combine a strong visual encoder with a strong language model, the resulting system can do things that neither component could do alone. It can reason about visual content — not just describe it, but make inferences, draw conclusions, connect visual observations to world knowledge. “What might happen next in this scene?” “What’s wrong with this code screenshot?” “Which of these interior designs would be more expensive to implement?”
This was the gap between CLIP (matching images to labels) and true visual language understanding (seeing, interpreting, and reasoning). GPT-4V closed it, and Gemini from Google quickly followed with its own natively multimodal architecture.
The Native Multimodal Approach
The LLaVA-style approach — gluing a vision encoder onto a language model — works but has an inherent limitation: the vision encoder and language model were trained separately, with different objectives, and the projection layer is a thin bridge between two worlds that don’t naturally align.5
Gemini (Google DeepMind, 2023) and its successors took a different approach: train a single model natively on interleaved multimodal data from the start. Text, images, audio, and video are all tokenized and fed into the same transformer. The model learns cross-modal associations during pretraining rather than acquiring them through a post-hoc alignment step.
The advantages of native multimodal training are significant. The model develops richer cross-modal representations because it sees the correlations between visual and textual features throughout training, not just during a fine-tuning phase. It can handle interleaved image-text inputs naturally — “here’s an image, some text about it, another image, a question about both” — without architectural gymnastics. And it scales more gracefully, because you’re training one model rather than stitching together two pretrained components.
The Gemma 4 family, released in early 2026, brought natively multimodal training to the open-weight ecosystem. All four variants — from the 2.3B effective parameter MoE up to the 31B dense model — accept text and image inputs natively. The 31B model handles tasks from visual question answering to document understanding to chart interpretation with remarkable fluency, and does so under Apache 2.0 licensing. As I wrote in my Gemma 4 analysis, this may be the most significant open-weight release since LLaMA.
Architectural Deep Dive: How Visual Tokens Enter the Transformer
The core engineering challenge of multimodal models is format conversion: images are 2D arrays of pixels; language models expect 1D sequences of tokens. Several approaches have emerged:
Patch embedding (ViT-style). Split the image into fixed-size patches (e.g., 16×16 pixels), flatten each patch into a vector, and project it to the transformer’s hidden dimension. A 224×224 image becomes 196 visual tokens. This is the approach used by Vision Transformer and, through the CLIP ViT encoder, most LLaVA-family models. Simple, effective, but expensive — 196 tokens per image adds significant context window cost.
Perceiver/Q-Former compression. Use a small set of learnable query tokens (e.g., 32 or 64) that cross-attend to the visual features and produce a fixed-size compressed representation. BLIP-2 (Li et al., 2023) used this Q-Former approach to reduce hundreds of visual tokens to ~32, dramatically reducing compute cost. The tradeoff is potential information loss — fine-grained visual details may be compressed away.
Dynamic resolution. Modern models like LLaVA-NeXT and Gemma 4 support variable image resolutions by splitting larger images into tiles and processing each tile separately. This preserves detail for high-resolution inputs (critical for OCR and document understanding) while remaining efficient for smaller images.
Native tokenization. The most radical approach: convert images directly into discrete tokens using a visual tokenizer (like a VQ-VAE) and feed them into the same vocabulary as text tokens. Chameleon (Meta, 2024) and Gemini use variants of this approach. The advantage is architectural simplicity — everything is just tokens. The disadvantage is that current visual tokenizers are lossy and may discard information that matters for fine-grained visual understanding.
Beyond Vision-Language
The term “multimodal” increasingly extends beyond vision and language. Current and emerging modalities include:
Audio. Whisper (OpenAI, 2022) demonstrated that a transformer trained on 680K hours of audio could achieve near-human speech recognition. Gemini natively processes audio alongside text and images. GPT-4o demonstrated real-time multimodal conversation with voice input and output, a capability that felt like science fiction when it was demoed in May 2024.
Video. Video understanding requires temporal reasoning — tracking objects across frames, understanding actions and their consequences, comprehending narratives that unfold over time. Models like Gemini 1.5 with its million-token context window can process hour-long videos by treating frames as image tokens. Video-LLaVA and VideoChat enable conversational interaction with video content.6
Code and structured data. This is less commonly discussed under “multimodal,” but language models that process code, tables, charts, and mathematical notation alongside natural language are multimodal in the relevant sense. The ability of Gemma 4 to look at a chart image and extract structured data from it — or to read a code screenshot and identify bugs — is a multimodal capability that depends on the interaction between visual processing and domain knowledge.
Robotics and embodiment. RT-2 (Google DeepMind, 2023) showed that a vision-language model could directly output robot actions — the model “sees” through cameras and “speaks” in motor commands. This connects multimodal AI to the physical world and opens the path toward general-purpose robots that can follow natural language instructions in novel environments.
What’s Working and What Isn’t
As of early 2026, multimodal models are remarkably capable on:
Image description and captioning. Essentially solved for natural images. Models generate accurate, detailed descriptions that often exceed average human effort.
Visual question answering. Strong performance on questions about image content, including spatial reasoning, counting (mostly), and commonsense inference.
Document understanding. OCR, chart interpretation, table extraction, receipt parsing — the combination of vision and language makes these practical for the first time.
Creative tasks. Describing, analyzing, and generating art; understanding and creating memes; visual storytelling.
They still struggle with:
Fine-grained spatial reasoning. “Is the red ball to the left or right of the blue box?” trips models up more often than you’d expect.7
Precise counting. “How many people are in this image?” is unreliable for counts above ~5, though performance is improving.
Temporal reasoning in video. Understanding what happens when in a video remains challenging, especially for long videos with complex narratives.
Multi-image reasoning. “What’s different between these two images?” or “Which of these five images best matches this description?” — tasks requiring comparison across multiple images — are harder than single-image understanding.
Compositional understanding. As I noted in the CLIP paper analysis, models still struggle with compositional relationships — understanding not just what is in an image but how the elements relate. “A dog chasing a cat” and “a cat chasing a dog” remain difficult to distinguish.
The Bias Problem
Multimodal models inherit and often amplify biases from their training data. This is not surprising — internet-scraped image-text data reflects internet demographics and stereotypes — but the consequences are more severe in the multimodal setting because the biases interact across modalities.
Agarwal et al. (2021) documented significant racial and gender biases in CLIP’s zero-shot classification. These biases propagated to downstream systems: Stable Diffusion generated images reflecting Western beauty standards and racial stereotypes; image search systems returned biased results for occupation-related queries.
The problem is harder to fix in multimodal systems because the biases are encoded at multiple levels: in the visual representations (what the model “sees” in images of different people), in the language representations (what associations the model has with descriptive terms), and in the cross-modal mapping (which images the model associates with which descriptions). Debiasing one modality is insufficient if the bias persists in the other.
GPT-4V’s system card documented extensive testing for bias and refusal to process certain image categories (identifying people by race, estimating age, etc.). This is a partial solution — it addresses the most obvious failure modes but leaves subtler biases in the model’s general visual understanding unaddressed. The field does not yet have a satisfying approach to multimodal fairness.
Where It’s Going
The trajectory is clear: multimodal is becoming the default. Every major frontier model released in 2025-2026 — GPT-4o, Claude 3.5, Gemini, Gemma 4 — is multimodal. Text-only models are increasingly niche. The question is no longer “should models handle images?” but “how many modalities can we integrate, and how deeply?”
I expect three developments in the next few years. First, unified models that handle text, images, audio, video, and code in a single architecture, trained end-to-end on all modalities simultaneously. Gemini is closest to this today. Second, generation across modalities — models that don’t just understand images but produce them, integrating diffusion models or similar generative components into the multimodal architecture. Third, embodied multimodal agents that perceive and act in the physical world, using tool calling and robotic interfaces to close the loop between perception and action.8
The deeper significance of multimodal AI is philosophical. Language models trained on text alone develop sophisticated capabilities — reasoning, planning, world knowledge. But they are, fundamentally, operating in a space of symbols divorced from the reality those symbols describe. Multimodal models begin to ground language in perception, connecting words to the visual, auditory, and physical world they refer to. Whether this grounding is “real understanding” or a more sophisticated pattern match is an open question — but it’s at least a step closer to the kind of integrated, embodied intelligence that humans have.