A research lab decides it is tired of bolting three different machines together to make one multimodal system. Instead of a diffusion model for images, a CLIP encoder for understanding, and a language model for text, it trains a single transformer to predict the next token — and lets that one trick produce and read images and video alike. That is the bet behind Emu3.
Key takeaways
- The Beijing Academy of Artificial Intelligence (BAAI / 智源研究院) released Emu3, a multimodal model family built to do both generation and understanding with one architecture.
- Emu3 is trained solely with next-token prediction, tokenizing images, text, and video into a shared discrete space and using a single transformer trained from scratch.
- It can generate images and video and understand images and video, without diffusion models or a CLIP-and-LLM combination.
- BAAI says Emu3 beats open models SDXL, LLaVA-1.6, and OpenSora-1.2 on their respective tasks.
- The 8B model and its vision tokenizer are open under Apache-2.0 on Hugging Face and GitHub.
- The paper, "Next-Token Prediction is All You Need," was posted in September 2024 and merged into Hugging Face Transformers in January 2025.
One model instead of a toolbox
The standard way to build a multimodal system in 2024 was to assemble specialists: a diffusion model to draw, a CLIP-style encoder to see, and a pretrained large language model (大模型) to talk. Each piece was its own trained network with its own failure modes. Emu3 throws that playbook out. BAAI treats images, text, and video as one thing — tokens — and trains a single transformer to predict the next one, the same objective that powers modern language models.
The paper's title says the ambition plainly: "Next-Token Prediction is All You Need." It is a direct nod to the "Attention Is All You Need" lineage, arguing that the same simple objective that scaled language can also scale vision and video if you first turn those modalities into a discrete token stream.
How it actually works
The mechanism is vector quantization. Emu3 uses a vision tokenizer that compresses an image (or video frames) into a grid of discrete tokens, the same kind of discrete units a language model already predicts. Text is already tokens. Video is just a sequence of tokenized frames. Once everything lives in one token space, a single transformer trained from scratch can be asked to continue any mixture of them.
That unification has a neat consequence: to make an image, the model simply predicts vision tokens until it has filled the canvas; to make a video, it predicts the next frame's tokens causally, the way a language model predicts the next word. To understand an image, it does the reverse — read the tokens and write a description. No separate diffusion sampler, no frozen CLIP, no pretrained LLM borrowed from elsewhere.
What it can do
BAAI demonstrates three jobs from one checkpoint family. Emu3-Gen produces images from text prompts at flexible resolutions and styles. Emu3-Chat reads images and answers questions about the physical world. And the video path can both generate short clips and, given a starting video, predict what happens next — a genuinely different approach from the diffusion-based video models like OpenSora that it is compared against.
The lab's headline claim is that this unified model outperforms established open specialists: SDXL on image generation, LLaVA-1.6 on vision-language understanding, and OpenSora-1.2 on video. Those are BAAI's selected comparisons, and they measure narrow tasks rather than all-around capability, but the fact that one from-scratch transformer can stand up to three purpose-built models is the result worth noting.
The claims and the catch
As always, the comparisons are the lab's own. "Outperforms SDXL, LLaVA-1.6, and OpenSora-1.2" is a statement about specific benchmarks the Emu3 team chose, not a verdict that Emu3 is universally better. Independent evaluators have not, to our knowledge, reproduced every head-to-head, so the fair reading is: BAAI has shown a single next-token model can match or beat three separate open models on their home turf — a structural result that is impressive even if the exact margins move once outsiders run the tests.
The deeper question is scaling. Emu3 is an 8B model, small by today's frontier standards, and the authors frame it as evidence that next-token prediction is a viable path to "general multimodal intelligence beyond language." Whether that path keeps paying off at 80B or 800B parameters is the open bet — and exactly the kind of thing the open weights are meant to let the community test.
Honest limitations
Emu3 is a research release, and its benchmark wins are BAAI's, not an independent lab's, so the "beats SDXL / LLaVA / OpenSora" line should be read as a claim to investigate, not settled fact. The unified next-token approach is elegant but still young: video length, resolution, and temporal coherence are constrained by the tokenizer and the compute needed to autoregressively generate many frames. Like every generative model, Emu3 can produce plausible-but-wrong images, captions, or video, and the open weights lower the barrier for both legitimate research and misuse. Nothing in the release suggests it should be treated as a source of factual truth about the world.
What readers can do now
- Pull the weights and run the three demos. Emu3 is on Hugging Face and GitHub under BAAI; try image generation, image chat, and video extension to see the unified design in action.
- Compare it on your own prompts, not the paper's. Run Emu3-Gen against SDXL and Emu3-Chat against LLaVA on tasks you care about; your domain will reveal limits the selected benchmarks hide.
- Watch the scaling story. Follow whether BAAI or the community trains larger next-token multimodal models; the interesting claim is not 8B parity but whether the single-architecture approach keeps working as it grows.
