Run the same workload at 16K and at 256K on the same H100, and the cost of every generated token rises by 3.6 to 7 times.
The figure comes from a 31-page preprint. Three qualifiers belong in the same breath as the number: it is a preprint, it has not been peer-reviewed, and all 12 authors work at Nvidia. The document is a research measurement, not a product plan. Drop any one of those and the finding reads as something it is not.
What the paper asks
The paper runs 13 benchmarks. The question is narrow: when a task has a long input and many steps, does the entire input have to sit in the context at once.
The method has two moves. Compaction: as the context nears its limit, earlier material is compressed into a summary that replaces the original text, freeing room to keep going. Selective retrieval: rather than loading every document, the system pulls only the passages the current step needs.
Together they keep the model working inside a short window. It sees compressed history and selected evidence instead of the full transcript.
Something is lost. Wording, tone, contradictions buried three sections apart — compression drops them. What to keep is itself a model judgment, and one bad judgment propagates through the following steps. That is the soft spot in the approach, and the authors do not hide it.
Accuracy held
The accuracy result is the part readers did not expect.
Constrained to a short working context, Claude Opus 4.5 and GPT 5.6 Sol lost very little accuracy. In some configurations they improved.
Improved is the word to stop on. A shorter context carries less distraction. A long window does not only cost more; it also places material in front of the model that has nothing to do with the step at hand. Window size and task difficulty are not two markings on one axis.
The paper's own summary: long inputs and long tasks do not automatically require long context.
The line item
The cost section is where the paper earns its citation.
On an H100, serving a 256K context against a 16K context raises cost per generated token by 3.6 to 7 times. The unit matters: generated tokens, not input tokens. A longer context means more prior text to attend to at every step, and compute and memory bandwidth rise with it.
Three models were tested: GLM 5.2, Kimi K2.6 and Qwen 3 Coder Next. Each produced its own number. The paper reports the range.
Why the range is the finding
Compressing that range into a single figure is the most common error in coverage of this work.
7 divided by 3.6 is about 1.94. The top of the range is nearly twice the bottom. That is the difference between one invoice and an invoice that doubles.
Say 7 times more expensive and you have taken the ceiling for the whole. Say 3.6 times and you have taken the floor for an average the paper never computed. Three models, three prices, one range — and which price applies depends on which model is actually in production.
The range is not an error bar. The price tag moves with the model.
Capacity is not price
16K to 256K is a 16-fold increase in window size. Cost per token rises at most 7-fold.
Laid side by side, those two multipliers invite a wrong conclusion: the window grew faster than the bill, so long context is a bargain. The error is putting two different quantities over the same denominator. Window size is capacity. Cost per token is price.
A larger window can hold more material, and a longer session can run more turns. At 3.6 to 7 times the unit price, more usage multiplies into a larger total, not a smaller one. The paper's own method runs the other way: keep the window small, retrieve what is needed, turn one long input into several short ones.
The exception the paper kept
The paper does not rule out tasks that genuinely need a long context.
It also does not enumerate them or estimate what share of workloads they represent. The 13 benchmarks do not cover that territory, and the authors do not pronounce on it.
So the useful order is: first ask whether the task requires seeing everything at once, then ask how wide to open the window. Reverse the order and the bill arrives before the answer. For tasks that do require the full text, compaction will lose something no matter how it is tuned. What compaction saves is the work that never needed the whole document.
The paper establishes that a large window does not need to be the default. It does not establish that a large window is useless. Between those two sentences sits an entire category of work.
What the price is attached to
Neither compaction nor retrieval is new. The contribution here is putting them into one measurement alongside cost, on one table.
Once they share a table, the choice becomes arithmetic. If accuracy drops a little after compaction, compact. If it drops a great deal, open the window and pay the 3.6 to 7 times.
That is the real weight of the paper. It does not overturn long context. It reclassifies long context from a capability ceiling into a price tier. The same task can be finished with a small window and compaction, or with one large window and one large bill. The difference is a trade between accuracy and cost, and at the margin, whether the task can be done in a single pass at all.
On one H100, with one set of weights, there are two ways to serve the same request and two prices. The gap is not set by the vendor. It is set by how many tokens are in the window.
*Disclaimer: This article is based on publicly available information and is for reference only. It does not constitute investment advice.*
