--- title: "Qwen Reaches One Million Tokens and Keeps Its Weights Open" date: 2026-10-29 category: Foundation Models site: NeuroAI canonical: https://neuroai.site/a/na-model-qwen-million-token-context language: en --- # Qwen Reaches One Million Tokens and Keeps Its Weights Open > Alibaba's Qwen team extended its open-weight models to a one-million-token context in January 2025, showing that long context can be a software problem, not just a hardware one. Long context is where most open models quietly fail. Alibaba's Qwen (通义千问) team treated it as an engineering puzzle—and solved it without asking for a bigger GPU. ## Key takeaways - On 27 January 2025, Alibaba open-sourced Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, the first open Qwen models to handle a 1,010,000-token context. - The leap came from Dual Chunk Attention (DCA), a length-extrapolation method that expands context without extra training, plus a sparse-attention inference framework. - The open inference framework processes one-million-token inputs roughly 3 to 7 times faster than a naive baseline. - Qwen's weights remain openly downloadable on Hugging Face and ModelScope under largely Apache 2.0 licences, keeping it among the most-used open model families. - In long-context benchmarks the 14B-1M variant beat both its own closed Qwen2.5-Turbo and OpenAI's GPT-4o-mini. ## The million-token problem A one-million-token context is about ten novels, 150 hours of a speech, or 30,000 lines of code in a single prompt. Most open models cap out at 128K tokens; push past that and attention breaks down because the model has never seen such distances between words during training. The naïve fix—train on longer sequences—costs a fortune in compute. Qwen's route was different. The team first trained Qwen2.5 progressively from a 4K context up to 256K, raising the RoPE base frequency from 10,000 to 10,000,000 to stabilise long-range position signals. Then, to reach 1M, they applied Dual Chunk Attention: a method that remaps the oversized relative distances between query and key into smaller, familiar values, so the model can generalise to lengths it never trained on. No further training required. ## Speed is the real bottleneck A million tokens is useless if the first reply takes ten minutes. Qwen's team built its inference framework on top of vLLM and added Minference-based sparse attention, chunked prefill and length-extrapolation optimisations. The result, the company reported, was a 3.2x to 6.7x speedup on prefilling million-token inputs across different model sizes and GPUs. The open framework itself was published on GitHub, so anyone can reproduce the deployment. In the "needle in a haystack" test at one-million-token length, the Qwen2.5-1M models retrieved hidden facts accurately, with only the 7B model showing minor errors. On harder long-context suites such as RULER and LV-Eval, the 14B-Instruct-1M beat both Qwen2.5-Turbo and GPT-4o-mini. ## Open weights as the default Qwen2.5, released in September 2024, ships in a wide size ladder from 0.5B to 72B under mostly Apache 2.0 licences; Qwen3 followed in April 2025. The 1M models continue that philosophy: weights on Hugging Face and ModelScope, an open serving stack, and a hosted Turbo option for those who would rather not run it themselves. The practical effect is that a developer can prototype on a 7B-1M open model locally, then scale to the 14B or call the API—without changing frameworks. That consistency is Qwen's quiet competitive edge. Many labs release one flashy model; Qwen releases a ladder, keeps the licences permissive, and maintains the tooling, which is what turns a download into a deployed product. ## Why context length is a deployment feature Long context is rarely a party trick; it is what makes certain products possible. A contract-review tool that must hold an entire 200-page agreement in memory, a code agent that reasons across a whole repository, or a research assistant that reads a book in one prompt all break down if the model forgets the start before reaching the end. Most teams work around short limits by chunking and retrieval, which adds latency, cost and a second failure mode—the retriever mis-ranks the relevant passage. Qwen's DCA approach is notable because it attacks the problem at the position-encoding level rather than by simply training longer, which keeps the method portable to future bases. The open sparse-attention framework is the other half: without the 3x–7x prefill speedup, the million-token context would be scientifically interesting but operationally unusable. That pairing—algorithmic reach plus systems engineering—is the same lesson DeepSeek demonstrated on cost, applied here to length. ## Honest limitations The 1M context, DCA method and speedup figures come from Alibaba's official Qwen blog and Alibaba Group's announcement; we have not re-run the benchmarks independently. The "3 to 7 times faster" range spans different hardware and model sizes, so your mileage will vary by GPU and sequence mix. Long-context accuracy degrades on tasks unlike the retrieval tests shown, and a 1M context does not mean the model reasons well across all of it—attention is necessary, not sufficient. Licensing is Apache 2.0 for most sizes but not all, so check the specific checkpoint. ## What readers can do now - For document- or repo-scale tasks, test Qwen2.5-14B-Instruct-1M before reaching for a far larger closed model—it often matches or beats GPT-4o-mini on long-context suites. - If you self-host, deploy with the open sparse-attention framework rather than a stock engine; the prefill speedup is the difference between usable and unusable at 1M tokens. - Always read the licence per checkpoint; "open-weight Qwen" is not a single licence, and the 3B and 72B sizes use a more restrictive terms than the rest. --- Published by NeuroAI (https://neuroai.site/) — https://neuroai.site/a/na-model-qwen-million-token-context Free to quote with attribution and a link to the original.