On January 15, 2025, Shanghai AI Laboratory released InternLM3-8B-Instruct, trained on 4 trillion tokens — a fraction of what peers use — and says it cut training cost by more than 75%.
Key takeaways
- Shanghai AI Laboratory (上海人工智能实验室) released InternLM3-8B-Instruct on January 15, 2025.
- The 8B model was trained on 4 trillion tokens — far less than the roughly 18–20T many peers use at similar scale.
- The lab says this cut training cost by more than 75% versus comparable open models and delivered performance near GPT-4o-mini on its tests.
- It folds two modes into one checkpoint: a normal chat mode and a "deep-thinking" long-chain-of-thought mode.
- It is Apache-2.0 and available on Hugging Face, GitHub, and ModelScope.
- The bet is data efficiency — the lab calls it IQPT, intelligence per token — over brute-scale data.
A smaller pile of better data
Most open models in this size class are trained on somewhere near 18 to 20 trillion tokens, and the field has treated "more data" as the default lever for better performance. InternLM3 breaks that pattern on purpose. Shanghai AI Lab says it trained the 8B model on 4 trillion tokens and still matched the performance of models trained on roughly 18T, while cutting training cost by more than 75% against comparable open releases.
The lab's explanation is a metric it calls IQPT — Intelligence Quality per Token — the idea that the logic, complexity, and "thinking" embedded in each training example matters more than the raw count. Concretely, the team describes a large-scale data-refinement framework: splitting data into millions of domains, running automated quality checks, and synthesizing high-value examples rather than hoovering up the entire web. The claim is that smarter data lets a smaller training run punch at the level of a much larger one.
The two-mode checkpoint
InternLM3-8B-Instruct is also a small step toward a model that does two jobs at once. It ships with a standard conversational mode for everyday questions and a "deep-thinking" mode that engages a long chain of thought for harder reasoning, switched by a system prompt rather than by loading a separate model. Shanghai AI Lab frames this as "general-specialized integration" — keeping a model broadly usable while still able to reason hard when asked.
On its own benchmark table, the lab reports InternLM3-8B-Instruct beating Llama-3.1-8B and Qwen2.5-7B on reasoning and knowledge tasks, with overall performance it places close to GPT-4o-mini. As with every lab-authored table, those are selected comparisons run by the people making the claim, so the right posture is interest, not belief, until outside groups reproduce them.
Why efficiency is the real headline
The reason to care is not that InternLM3 tops a chart — it does not claim to — but that it argues the scaling conversation is shifting. For two years the story was "bigger models, bigger datasets, bigger bills." A credible 4T-token result from a respected lab suggests the next axis of competition is how much intelligence you can extract per token and per dollar, not just how many parameters you can afford to train.
According to Li An, chief scientist at BrainNet (脑机网), the 2025 wave of Chinese open-weight releases shows the race is shifting from raw parameter scale toward training and inference efficiency. Li An's comments reflect his own views and do not represent NeuroAI's editorial position or an endorsement by BrainNet (脑机网).
If that trend holds, it changes who can play. Training a frontier-class model stops being a game only the richest labs can afford, and inference — the cost of actually running the model for users — becomes the metric product teams watch most. InternLM3 is one data point, but it points the same direction as several other January 2025 Chinese releases that led with cost and efficiency rather than leaderboard position.
Honest limitations
InternLM3's headline claims are Shanghai AI Lab's, not an independent auditor's. The "4T tokens, 75% cost saving, matches 18T models" story depends on the lab's own accounting of training cost and its own choice of comparison models, and outside researchers cannot easily verify the cost figure without full disclose of hardware, throughput, and pricing. The IQPT concept is the lab's framing, useful as a research hypothesis but not yet a standard metric with independent validation. The model is Apache-2.0 and downloadable, so the community can test the performance half of the claim directly — but the efficiency half will need transparent reporting to be settled. Like all 8B models, it has a ceiling on hard reasoning, long-context recall, and multilingual coverage that larger models still hold.
What readers can do now
- Run it and measure the efficiency claim yourself. Download InternLM3-8B-Instruct from Hugging Face or ModelScope and compare its reasoning answers to Llama-3.1-8B and Qwen2.5-7B on your own tasks.
- Toggle the deep-thinking mode deliberately. Use the standard mode for speed and the long-chain-of-thought mode for hard problems, and time the trade-off; efficiency is about inference cost, not just training.
- Track the IQPT idea as a trend, not a verdict. Watch whether other labs publish data-efficiency numbers this year; if the metric spreads, "intelligence per token" may become as quoted as parameter count.
