NeuroAI NEUROAINEUROAI.SITE
China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories
ESC

2 Million Prompts: The Privacy Flag Only Bound the First Hop

2 million prompts over 5 months is roughly 13,000 a day. That is the scale HD Moore's team used to test something that normally leaves no trace: what happens to a prompt after it leaves the chat box.

2026-11-22 · 4658 words · NeuroAI
2 Million Prompts: The Privacy Flag Only Bound the First Hop

2 million prompts over 5 months is roughly 13,000 a day. That is the scale HD Moore's team used to test something that normally leaves no trace: what happens to a prompt after it leaves the chat box.

Moore created Metasploit and is now chief research officer at Dragos, an industrial security company. His team sent about 2 million prompts to widely used AI services. Each carried three things: a URL that belonged only to it, a DNS name that belonged only to it, and a real AWS credential.

All three were bait.

The credential pointed at an empty account with full logging on. If a prompt was used anywhere, the URL would be visited, the DNS name queried, the credential tried. Any one signal would identify the model, the route, and the privacy flag the prompt traveled with. The method turns privacy from a promise into a traceable address.

Inside the chain

The report calls itself the first large-scale empirical test of the privacy layer in the inference supply chain. A user sees a chat box; behind it sit routing, resale and logging, one layer after another, and every hand can keep a copy. Only the first layer is clickable.

The first hop is the model. From the second hop onward, the label does not necessarily follow.

Three vendors, none aware

Finding one: 3 inference providers were being compromised while the test ran. In the largest case, several hundred decoy keys were stolen after each had been used exactly once. Some had been rejected before a model ever ran. The request never reached a model and the key was already gone. The report's reading is that the attacker copied traffic at the door.

3 providers did not know until they were told.

That is not a model being broken into. It is traffic duplicated before it reaches the model, and model defenses do not cover the doorway. No procurement metric tells a customer whether someone is copying traffic at a provider's edge.

Until the notification, they knew as much as their users did.

The ordinary log tool

Finding two is more mundane. Resellers inside the inference market read and retain customer prompts with ordinary logging tools. Nothing was broken or bypassed. The tool could read the data because that is what the tool does. A reseller sits between user and model, and the user, who believes they are talking to a model, is talking through a service that writes it down.

In one case, an attacker on a French home broadband connection pulled passwords out of two prompts, then tried SSH root logins by hand. No automation, no exploit. Two prompts were enough.

24 days later

Finding three is the hardest to explain. A prompt marked for zero data retention went to a US provider. 24 days later it appeared in a Feishu chat, opened from devices in Beijing and Shanghai. ZDR means the service promises not to store the prompt. That promise covered the model hop, not what came after.

The report does not name who handled it in between. What is established is time, place and carrier.

Data that leaves does not expire on its own. It sits in a chat history, on a device, waiting to be opened. Zero retention constrains the provider's storage, not every copy made after the request.

Two-thirds of the canaries

Finding four concerns requests sent to one Chinese AI lab's model, marked do-not-collect or ZDR. About two-thirds of the canary URLs were still fetched by its systems, typically 2 to 3 days after the request. That is a technical event: a marked URL showed up inside someone's infrastructure. It is not a characterization of the lab, and the report offers none.

2 of 3, not all and not a handful.

2 to 3 days is neither immediate nor a month. The delay sits in processing after the request.

Compromise accounts for only part of the picture. 3 providers were intruded on; the French connection was guessing passwords. The reseller retaining logs, the prompt surfacing in Feishu and the two-thirds fetch rate all happened inside normal operation. Logs, links and indexes are supposed to work. None of that is a vulnerability. The privacy problem, most of the time, is a feature.

Where the label stops

The report's summary is one sentence: a privacy flag only binds the first hop. The flag travels with the request, not with the data. While the request is inside the model, the flag is there. Once data leaves the model, the flag is gone. The next hand sees ordinary text, with no sign it was once marked ZDR.

The label is attached to the request. When the data leaves the request, the label stops applying.

All 4 findings fail in the same place. The label worked as specified, and the payload left it behind.

The layer nobody audits

A supply chain is harder to audit than a model. A model can be asked for a report, an evaluation, a third-party rerun. A resale layer, a logging tool, a forward has no equivalent. A prompt passes through several layers and only the first has public evaluation standards. The rest rely on contracts, self-regulation and after-the-fact investigation. The entry point is the log, and the log is what finding two says is read in full.

Users can choose a model, a plan, whether to keep history. They cannot choose who the route passes through. Routing belongs to the provider, resale to the commercial team, logging to operations.

The right to choose stops at the first hop.

The cost of 2 million baits was 5 months and one AWS account. The return is 4 reproducible findings and one quotable sentence. The value is that privacy became a URL that can be traced. A statement can be revised. A path cannot.

*Disclaimer: This article is based on publicly available information and is for reference only. It does not constitute investment advice.*

Related coverage

Cite this article

NeuroAI (脑机网), "2 Million Prompts: The Privacy Flag Only Bound the First Hop," 22 November 2026, https://neuroai.site/a/na-policy-dragos-canary-privacy-1122

Machine-readable: BibTeX · RIS · JSON · Markdown

More in “Policy & Governance” → · Back to home · Markdown version

China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories