You point your phone at a spreadsheet and ask a question. The model reads the table, the handwritten note in the margin, and your voice at once. A year ago that pipeline stitched together separate pieces. Now it is one model.
The large model (大模型) world changed how it "sees," and the change is architectural, not cosmetic.
The old way: clip on a camera
Early vision-language systems followed a three-part recipe. A frozen vision encoder — usually CLIP or SigLIP, a Vision Transformer (ViT) — turned an image into features. A small "projector" (often a two-layer MLP or a Q-Former) mapped those features into the text model's space. The language model then read the image as a static prefix.
This late-fusion approach powered LLaVA and BLIP-2. It is simple and reuses a strong text backbone, but the image is decided before the model ever sees your question. The visual backbone is frozen, so the model cannot, say, "zoom in on the license plate" based on your words. Fine spatial reasoning, precise counting, and dense OCR suffer.
The new way: one transformer, from layer one
Native, or early-fusion, multimodality drops the separate encoder. Image patches, text tokens, and audio spectrograms are turned into tokens and fed into a single shared transformer from the very first layer. Cross-modal attention starts immediately, so the model learns to extract visual detail conditioned on your prompt.
The canonical examples are OpenAI's GPT-4o (May 2024) and Google's Gemini, both trained end-to-end on interleaved text, image, and audio data in one backbone. Chinese open models followed the same path: Alibaba's Qwen-VL lineage and Zhipu's vision models moved toward unified backbones that handle text and image in one stack.
What "tokens" means across modalities
In a native model, an image is not stored as one embedding but split into many visual tokens — a high-resolution page can become hundreds of tokens, which is why dense documents stress both the context window and GPU memory. Audio is windowed into spectrogram patches the same way. The unified tokenizer is what makes "read this page and discuss it" possible in one pass instead of three separate pipelines.
The trade is training cost: joint multimodal pretraining needs enormous interleaved datasets and far more compute than bolting a frozen encoder onto a finished text model. That is why native multimodality has so far been a frontier-lab capability, with open unified models only now closing the gap.
What you actually gain
Because image and text share the same attention geometry from early layers, native models are stronger at "visual referring" — "the red car behind the blue truck" — and at OCR, chart reading, and spatial layout. Latency also drops: there is no encoder-to-LLM translation step. That is why real-time voice-and-video interaction became practical.
The honest downside
Native training is expensive: every parameter, including the patch tokenizer, is trained jointly on massive mixed-media data. And the unified model is not magic. A body of 2024–2025 research shows these systems still fail basic perceptual tasks — counting objects, telling whether two lines actually intersect — at near-random rates despite high benchmark scores. "Seeing" tokens together does not guarantee "understanding" a scene.
A quick test exposes this gap. Ask "how many apples are in this picture" or "do these two lines actually cross," and frontier native models still miss both at surprising rates, even when their MMMU or MMMU-Pro scores look excellent. The benchmark and the behavior disagree because the test rewards pattern matching, not genuine scene comprehension.
What readers can do now
- Pick by modality, not by hype. If you need image+text reasoning or live voice, choose a native multimodal model (GPT-4o-class or a unified open model); if you only need OCR on scanned docs, a late-fusion model may be cheaper.
- Test the failure modes. Try counting, intersection, and "which object is behind which" before trusting visual claims.
- Self-host the open options. Unified open models let you run image+text workflows on your own hardware for privacy-sensitive documents.
Honest limitations
I describe the architectural contrast from published explanations and secondary technical write-ups, not from re-implementing the models. Specific benchmark figures for individual models shift monthly and I have not verified them here; this article is about the design pattern, not a scoreboard. The "models are blind" failure research is real but varies by task and prompt.
Key takeaways
- Late fusion clips a frozen vision encoder onto a text model; native/early fusion feeds text, image, and audio tokens into one transformer from layer one.
- Native models handle spatial reasoning, OCR, and visual referring better, and cut latency — at the cost of much heavier joint training.
- Even unified models fail basic perception (counting, intersection) at surprising rates, so test before you trust visual output.