--- title: "Shanghai AI Lab's 8B model quietly rewrote the math on training cost" date: 2026-10-28 category: Foundation Models site: NeuroAI canonical: https://neuroai.site/a/na-model-internlm3-open-llm language: en --- # Shanghai AI Lab's 8B model quietly rewrote the math on training cost > InternLM3-8B-Instruct from Shanghai AI Laboratory was trained on 4 trillion tokens and the lab says it cut training cost by over 75% versus comparable models, while giving the weights away under Apache 2.0. Most labs still act as if you need more data to get a smarter model. A state-backed lab in Shanghai built an 8-billion-parameter model on a fraction of the tokens its peers used — and then gave the weights away for anyone to run. The bet is that data quality, not data volume, is the real bottleneck. InternLM3-8B-Instruct, released by Shanghai AI Laboratory (上海人工智能实验室) on 15 January 2025, is the third generation of the 书生·浦语 (InternLM / Shusheng Puyu) family. The lab's headline claim is simple to state and hard to ignore: the model was trained on only 4 trillion high-quality tokens and, by the lab's accounting, cost more than 75% less to train than other large language models (大模型) of similar scale, while matching or beating Llama-3.1-8B and Qwen2.5-7B on reasoning and knowledge tasks. ## Key takeaways - InternLM3-8B-Instruct is an 8-billion-parameter dense model, open-weight under the Apache 2.0 license, with weights on Hugging Face, ModelScope, and the lab's own channels. - The lab reports training on 4 trillion tokens with a "data refinement" pipeline that lifted what it calls "thinking density," cutting training cost by over 75% versus comparable models. - It ships two modes in one model: a deep-thinking mode that emits a long chain-of-thought for hard problems, and a normal mode for fluent chat, switchable with a system prompt. - On OpenCompass evaluations the lab reported results close to GPT-4o-mini, a vendor claim rather than an independent audit. - It remains the last dense, text-only, general-purpose LLM in the family; later Intern-S and Intern-S2 releases moved toward multimodal scientific reasoning. ## The efficiency argument The number that matters is not 8 billion parameters but 4 trillion tokens. Mainstream open models at this size are typically trained on far more data; the lab argues that smarter filtering and synthetic data generation let it reach the same capability with less. If the accounting holds, it changes the economics of who can train a competitive base model. You no longer need a hyperscaler's corpus to field a useful 8B model — you need a disciplined data pipeline. The mechanism the lab describes is agentic self-evolution: automated quality checks scrub noise across millions of domain zones, and a "general-to-special" synthesis loop generates high-value training data. The result the lab points to is a smaller, cheaper-to-train model that still lands near stronger proprietary systems on the lab's own OpenCompass benchmark suite. ## Why two modes in one model Earlier designs needed a separate "reasoning" model and a separate "chat" model. InternLM3 folds both into one checkpoint. A system prompt flips between a normal response for everyday conversation and a deep-thinking mode that shows its long chain-of-thought for math, code, and planning. For a developer, that means one model to host instead of two, with a clear trade-off: deeper answers cost more latency. According to Li An, chief scientist at BrainNet (脑机网), the lasting value of InternLM3 is less the benchmark rows and more the proof that a public research institute can publish a reproducible, low-cost training recipe the wider community can actually reuse. ## Position in the open ecosystem InternLM is one of China's longest-running open LLM lines, developed with SenseTime, the Chinese University of Hong Kong, Fudan University, and Shanghai Jiao Tong University. The 8B model runs on a single consumer GPU in quantized form, which is why it shows up in private-deployment and education scenarios where data cannot leave the building. Its successors — Intern-S1, S1-Pro, S2-Preview, and S2-Mobius — are larger multimodal scientific models, so the 8B Instruct remains the family's last small, general-purpose, text-only base. ## Honest limitations The 75% cost reduction and the benchmark comparisons are the lab's own statements; no independent reproduction of the training-cost figure was found across two authoritative sources, so treat it as a vendor claim, not a verified constant. The "near GPT-4o-mini" result comes from the lab's OpenCompass runs, not a third-party leaderboard. Context length for the base 8B is described inconsistently across databases (one lists about 128K tokens, others cite long-context variants), so no single number is asserted here. As with any open-weight release, safety behavior depends on how downstream users deploy it. ## What readers can do now - If you need a small, permissively licensed base model for Chinese and English tasks, pull InternLM3-8B-Instruct and benchmark it against Qwen2.5-7B and Llama-3.1-8B on your own eval set. - Run it locally with LMDeploy, vLLM, Ollama, or llama.cpp in 4-bit or 8-bit to see if single-GPU deployment fits your budget. - If you train your own models, study the lab's data-refinement framing: the 4T-token recipe is the part worth copying, not the parameter count. --- Published by NeuroAI (https://neuroai.site/) — https://neuroai.site/a/na-model-internlm3-open-llm Free to quote with attribution and a link to the original.