NeuroAI NEUROAINEUROAI.SITE
China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories
ESC

Self-Hosted Inference: 40% Cheaper, 80% Faster to First Token

Cut time to first token by 80% and 20% remains. The wait falls by a factor of 5.

2026-11-22 · 4868 words · NeuroAI
Self-Hosted Inference: 40% Cheaper, 80% Faster to First Token

Cut time to first token by 80% and 20% remains. The wait falls by a factor of 5.

That figure comes from AWS and its distributed KV cache, alongside a cost number: self-hosted LLM inference up to 40% cheaper. $10 of inference becomes $6 in the best case. Both describe the best-performing configuration, not the average bill across every self-hosted cluster. Treat them as averages and the first invoice will not reconcile.

Neither number is an average.

A model answers in two stages: it reads the prompt, then writes. Reading is prefill, writing is decode, and time to first token measures the first stage. In a multi-turn conversation, every turn rereads the history, and long system prompts and many attachments make that read more expensive. Most of what a user experiences as slow happens here.

20 turns means the first 19 are read 20 times.

Nothing in that prefix changed. The same tokens were computed last turn and are computed again this turn. A cache stores the intermediate results, not the original text.

Where the cache lives

One card's memory cannot hold a long cache, which is the first wall. The longer the context, the thicker the intermediate state, and a single card fills first. Distributed KV caching moves those results into another storage tier and fetches them back on the next turn. Fetching is cheaper than recomputing, and the saving is that round trip.

The saving depends on hits. With no hits, it is zero.

That determines the character of the 40%. It is a maximum, for the configuration with the best hit rate, the most repeated prefixes and the steadiest load. Change the workload on the same machine and the saved percentage changes with it.

The ceiling, not the average

A 40% cut leaves 60% of the cost, so the same budget runs 1.67 times the volume. One divided by 0.6 is 1.67, and that holds only where the 40% holds.

1.67 times is also a ceiling.

The logic favors long context most. A 100,000-word document, a conversation dozens of turns deep, an entire codebase: the prefix repeats many times, and the more it repeats the more compute the cache removes. A one-off long input that differs every time gets no help. Caching does not make long context cheap. It makes repeated long context cheap.

Length is the cost. Repetition is the discount.

Cost and latency usually trade against each other. Raise the batch size and throughput climbs while the first token slows. Caching removes duplicated computation rather than changing parallelism, so both can fall together.

Peak traffic, a cold cache and prefixes that share nothing, all at once, and the 40% may not appear at all. Plan capacity for the worst day.

Orchestration changes the unit

The same update included two items from AWS Quick: conversational orchestration of multi-agent workflows, and persistent tasks on the web. Both stretch a single request into a sustained run. Multi-agent work divides labor turn by turn. A persistent task keeps going with nobody watching, and because the task does not finish, the resources it holds are not released.

From one request to one stretch of time.

Longer processes repeat more. Each agent carries its own role description and tool list, reread every round. A persistent task wakes up each day and reads the same background first. Orchestration splits a request into many calls; caching collects what those calls share.

When a request becomes a process, the billing unit changes. A per-call bill cannot describe a task that runs for 3 days and waits for a human in the middle.

Idle is the line item

Persistence has a side effect. Resources stay occupied while nothing happens. Cache, session and context wait for someone to return, and one stalled agent stops the whole flow without releasing the memory it holds. Every idle minute is another minute of cost, written not on the request but in the utilization of the machine.

Self-hosting has at least three lines: compute, cache, idle. The first is easy. The last two are guesses.

Idle capacity is unavoidable once you own the hardware. Capacity is provisioned for the peak, and after the peak the cards are still there. Metered cloud billing spreads idle capacity across other customers; self-hosting keeps it. That cost is not in the 40%, but it is in the invoice.

Pricing power moves down a layer

For a while, the list price of inference was set by the model maker. An endpoint went up, a price per million tokens was posted, and everyone bought at it. Self-hosting splits the line apart. The same weights, on whose cards, with what cache configuration, batched how, produce different unit costs. Price turns from a list number into an engineering problem. A list price can be copied. Engineering cannot.

Whether self-hosting pays depends on the shape of the load.

Steady, high-frequency traffic with repeated prefixes fills the cache. Traffic that swings has to buy cards for the peak anyway.

Updates like this come from cloud providers, not model makers. Model makers sell intelligence; cloud providers sell the efficiency of running it. A lower barrier is not the same as no staff: clusters need operations, caches need tuning, capacity needs planning, and none of those three is inside the 40%.

Budget for the worst day

Take the best case from a launch post and use it to budget the worst case. The 40% is a ceiling on savings, not a commitment, and the 80% comes from the same best-case set. Budget to the ceiling and the first invoice forces a recalculation.

Both numbers work as a signpost, not a forecast. They say caching is worth doing, not how much it will save you. That number has to come from your own traffic.

When the cache hits, the prefix computed last turn is not computed again. Both percentages come out of that one sentence.

*Disclaimer: This article is based on publicly available information and is for reference only. It does not constitute investment advice.*

Related coverage

Cite this article

NeuroAI (脑机网), "Self-Hosted Inference: 40% Cheaper, 80% Faster to First Token," 22 November 2026, https://neuroai.site/a/na-chips-aws-kv-cache-cost-1122

Machine-readable: BibTeX · RIS · JSON · Markdown

More in “Chips & Compute” → · Back to home · Markdown version

China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories