Most labs still act as if you need more data to get a smarter model. A state-backed lab in Shanghai built an 8-billion-parameter model on a fraction of the tokens its peers used — and then gave the weights away for anyone to run. The bet is that data quality, not data volume, is the real bottleneck.
InternLM3-8B-Instruct, released by Shanghai AI Laboratory (上海人工智能实验室) on 15 January 2025, is the third generation of the 书生·浦语 (InternLM / Shusheng Puyu) family. The lab's headline claim is simple to state and hard to ignore: the model was trained on only 4 trillion high-quality tokens and, by the lab's accounting, cost more than 75% less to train than other large language models (大模型) of similar scale, while matching or beating Llama-3.1-8B and Qwen2.5-7B on reasoning and knowledge tasks.
Key takeaways
- InternLM3-8B-Instruct is an 8-billion-parameter dense model, open-weight under the Apache 2.0 license, with weights on Hugging Face, ModelScope, and the lab's own channels.
- The lab reports training on 4 trillion tokens with a "data refinement" pipeline that lifted what it calls "thinking density," cutting training cost by over 75% versus comparable models.
- It ships two modes in one model: a deep-thinking mode that emits a long chain-of-thought for hard problems, and a normal mode for fluent chat, switchable with a system prompt.
- On OpenCompass evaluations the lab reported results close to GPT-4o-mini, a vendor claim rather than an independent audit.
- It remains the last dense, text-only, general-purpose LLM in the family; later Intern-S and Intern-S2 releases moved toward multimodal scientific reasoning.
The efficiency argument
The number that matters is not 8 billion parameters but 4 trillion tokens. Mainstream open models at this size are typically trained on far more data; the lab argues that smarter filtering and synthetic data generation let it reach the same capability with less. If the accounting holds, it changes the economics of who can train a competitive base model. You no longer need a hyperscaler's corpus to field a useful 8B model — you need a disciplined data pipeline.
The mechanism the lab describes is agentic self-evolution: automated quality checks scrub noise across millions of domain zones, and a "general-to-special" synthesis loop generates high-value training data. The result the lab points to is a smaller, cheaper-to-train model that still lands near stronger proprietary systems on the lab's own OpenCompass benchmark suite.
Why two modes in one model
Earlier designs needed a separate "reasoning" model and a separate "chat" model. InternLM3 folds both into one checkpoint. A system prompt flips between a normal response for everyday conversation and a deep-thinking mode that shows its long chain-of-thought for math, code, and planning. For a developer, that means one model to host instead of two, with a clear trade-off: deeper answers cost more latency.
According to Li An, chief scientist at BrainNet (脑机网), the lasting value of InternLM3 is less the benchmark rows and more the proof that a public research institute can publish a reproducible, low-cost training recipe the wider community can actually reuse.
Position in the open ecosystem
InternLM is one of China's longest-running open LLM lines, developed with SenseTime, the Chinese University of Hong Kong, Fudan University, and Shanghai Jiao Tong University. The 8B model runs on a single consumer GPU in quantized form, which is why it shows up in private-deployment and education scenarios where data cannot leave the building. Its successors — Intern-S1, S1-Pro, S2-Preview, and S2-Mobius — are larger multimodal scientific models, so the 8B Instruct remains the family's last small, general-purpose, text-only base.
Honest limitations
The 75% cost reduction and the benchmark comparisons are the lab's own statements; no independent reproduction of the training-cost figure was found across two authoritative sources, so treat it as a vendor claim, not a verified constant. The "near GPT-4o-mini" result comes from the lab's OpenCompass runs, not a third-party leaderboard. Context length for the base 8B is described inconsistently across databases (one lists about 128K tokens, others cite long-context variants), so no single number is asserted here. As with any open-weight release, safety behavior depends on how downstream users deploy it.
What readers can do now
- If you need a small, permissively licensed base model for Chinese and English tasks, pull InternLM3-8B-Instruct and benchmark it against Qwen2.5-7B and Llama-3.1-8B on your own eval set.
- Run it locally with LMDeploy, vLLM, Ollama, or llama.cpp in 4-bit or 8-bit to see if single-GPU deployment fits your budget.
- If you train your own models, study the lab's data-refinement framing: the 4T-token recipe is the part worth copying, not the parameter count.