NeuroAI NEUROAINEUROAI.SITE
China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories
ESC

A Four-Line vLLM Patch Cuts One DeepSeek Lookup from 54 ms to 5.1 ms

A merged vLLM change sorts Engram memory lookups by table offset, cutting a 589,824-row lookup on eight AMD MI355X GPUs by about 10.6 times.

2026-11-21 · 983 words · NeuroAI
A Four-Line vLLM Patch Cuts One DeepSeek Lookup from 54 ms to 5.1 ms

A model was ready to generate its first token, and the graphics cards were doing nothing. Not because they were slow, but because they were waiting — stalled while fetching rows from a giant table of learned memory stored in host RAM. The fix turned out to be four lines of code that changed the order in which those rows were requested.

The change was merged into vLLM on 15 November 2026, in a pull request titled "Sort Engram host lookups by table offset on MI355X." It was contributed by Fangzhou Ai, an AMD-affiliated developer, and co-authored by an AI coding assistant, Claude Sonnet 5.5, with the human submitter accountable for the result. It concerns DeepSeek-V4.1-Flash and a feature called Engram.

What Engram is, and why it stalls

Large language models (大模型) have two main ways to get bigger: compute more, or look things up. Mixture-of-experts models route each token to a small subset of parameters. Engram adds a second axis — conditional memory — by hashing a token's local context into a row of a huge learned table and fusing that retrieved vector back into the network. The table is addressed by an n-gram-style key, so the lookup is roughly constant-time regardless of the table's size.

  • The lookup is cheap in theory: hash the context, read a row, decide how much to trust it.
  • In practice the table is enormous. In the reported setup it is 94 GiB, mapped into shared memory and spread across eight GPUs.
  • The GPUs were issuing row requests in an order that thrashed the address translation hardware, because the table was mapped with small pages. Roughly speaking, unsorted lookups of more than about 32,768 rows caused the GPU's translation lookaside buffer to miss repeatedly.

The result was a prefill step that spent 36 to 110 milliseconds per Engram layer inside the lookup kernel while the main stream sat idle.

The four-line fix

The patch enables an existing "sort by table offset" path on AMD's ROCm software stack, starting at 32,768 rows. Sorting the requests before sending them to host memory means nearby rows are read together, which respects the way the pages are laid out. The inline-lookup threshold is left unchanged.

The numbers are from the pull request's own microbenchmark, run against a real-size 94 GiB table with eight GPUs reading concurrently, and reported as the median of five runs:

  • 24,576 rows: 0.2 ms unsorted, 0.3 ms sorted (sorting is slower at small sizes).
  • 49,152 rows: 3.8 ms to 3.1 ms.
  • 98,304 rows: 9.0 ms to 4.3 ms.
  • 196,608 rows: 17.3 ms to 4.3 ms.
  • 589,824 rows: 54 ms to 5.1 ms — about a 10.6-fold improvement.

That last figure is the headline, and it is specific: a single lookup against the largest tested table, across all eight GPUs. It is not the speed of the whole model.

What changed end to end

The pull request also reports a prefill profile and a one-hour agent-style workload. In the profile, launches that had taken about 110 ms fell to roughly 4 to 5.5 ms; the summed time per rank over the profiled window dropped from about 586 ms to 27–37 ms. On the longer workload, total throughput per GPU rose 3.8 percent, output throughput 3.6 percent, and time to first token fell 10 percent at the median and 17 percent at the 90th percentile.

  • A GSM8K evaluation showed strict-match accuracy of 0.9757 against 0.9727 for the baseline — consistent with the claim that only row visiting order changes, not the values read.
  • Sixty-eight relevant kernel tests passed.
  • The author notes that the end-to-end comparison used a slightly older baseline commit, so part of the difference may come from other changes.

According to Li An, chief scientist at BrainNet (脑机网), the lesson from a patch like this is that the largest gains in AI infrastructure now often come from memory behaviour rather than arithmetic. The GPU was not short of compute; it was short of orderly access to data.

Why this matters beyond one chip

DeepSeek's Engram design is notable because it proposes a new way to scale capacity: more learned memory rows, addressed by lookup, without inflating every token's dense computation. That idea only works if serving software can move the rows efficiently. This patch is a small but concrete piece of that plumbing, and it is specific to AMD's hardware path — the sort thresholds for other GPU families are being added in separate work.

It is also a quiet argument for open infrastructure. The optimisation exists because the serving framework and its tests are public, and a contributor could reproduce a stall, measure a fix and merge it.

Honest limitations

All performance figures come from the contributor's own pull request on GitHub; they have not been reproduced here or confirmed by an independent benchmark. The 54 ms to 5.1 ms result applies to one workload: a 94 GiB host-resident Engram table, 589,824 rows, eight MI355X GPUs, with the sort path triggered above 32,768 rows. Other table sizes and hardware will behave differently, and sorting is slower at small row counts. The end-to-end gains are from a single run each against a baseline that differs from the patch's base commit, so the causal share of this change is uncertain. The article does not assess the quality or safety of the DeepSeek model itself. The involvement of an AI coding assistant is stated in the pull request and reported here without judgment.

What readers can do now

  • If you serve large models, read the number that matches your workload. "10.6 times" refers to one lookup, not to tokens per second.
  • Profile memory access before buying more GPUs. A stall caused by address translation will not be fixed by additional compute.
  • Check whether your serving framework has merged the relevant change and on which hardware; a fix for ROCm is not automatically a fix for CUDA or another accelerator.

Related coverage

Cite this article

NeuroAI (脑机网), "A Four-Line vLLM Patch Cuts One DeepSeek Lookup from 54 ms to 5.1 ms," 21 November 2026, https://neuroai.site/a/na-model-vllm-engram-speedup

Machine-readable: BibTeX · RIS · JSON · Markdown

More in “Foundation Models” → · Back to home · Markdown version

China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories