NeuroAI NEUROAINEUROAI.SITE
China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories
ESC

The Agent Can Click Now. The Permission Boundary Isn't Drawn.

A window sits behind another window on a Mac. Nothing is in front of it.

2026-11-22 · 4620 words · NeuroAI
The Agent Can Click Now. The Permission Boundary Isn't Drawn.

A window sits behind another window on a Mac. Nothing is in front of it. Something inside it is being clicked.

Google says Gemini can do this, and calls it a model-side ability, not new hardware. The machine and the interface are unchanged. What changed is who is operating it.

Tencent's WorkBuddy shipped a Computer Use feature with four claims in its public description. It can recognize the current page. It can understand a client application's interface. It can click, type and scroll. It does not need the target software to open an API.

Three claims about what it can do. One about what it skips.

An API is a door with a lock

When software talks to software, it goes through an interface. The interface is a door, and the vendor holds the key. The key states which doors it opens, how many times a day, and whether it can bring anyone else in.

To use it, you knock, identify yourself, and wait for the vendor to configure access, which it may grant in part or not at all. The documentation says what can be read, what can only be written, and what an error returns.

That process is slow. It is also where responsibility gets written.

A screen has no door

A screen is not an interface. It holds only what has been drawn: buttons, fields, menus, dialogs. These are arranged for eyes, not for calls.

To read a screen, an agent has to recognize that a shape is a button, then judge whether that button can be pressed right now. If it misreads the shape, the click still goes out.

Nothing on that path is locked: the vendor was never involved and issued no permission. What the agent holds is not a key with a scope written on it. It is a surface that displays and accepts clicks. Whatever is on the surface is reachable.

Reachable is not correct. Some clicks can be undone. Some cannot. An interface documents which command is dangerous. A screen does not. A screen shows a button and nothing about what follows.

Who gave the consent

On the interface path, consent comes from the software vendor, written into the documentation.

On the screen path, the public description does not say who gives it. Three of the four claims describe ability. One says no interface is required. None describes an authorization: who opened it, how wide, how to close it.

That absence is not proof that no design exists. A product may have an internal permission model that has not been published. Or the permission may simply be that the program was installed on this machine. Both are consistent with what has been said, and the public record cannot separate them. One version has a switch. The other has only the act of installation.

Two kinds of background

Background can mean two things.

The first is a window that is not in front. Other software is on screen, the working window sits underneath, and the clicks still land.

The second is a process running unattended, with nobody watching at all.

Both point at one condition. When the action happens, a person may not be looking. In the first, the person can switch windows and see it. In the second, they may never know it ran.

That move changes the human's job from operator to inspector, and inspection happens after the fact, when the cost has already been paid. There is also no hand in the frame: a person clicking leaves motion, a program clicking leaves only a result.

Reading a page is not reading an interface

Two of the four claims are separate. Recognizing a page and understanding a client interface are different problems.

A page has structure: what sits above what, and what can be clicked, is written into the markup. A client application often offers only pixels: a button is a patch of color, a field is a line.

Reading structure fails loudly. Recognizing pixels fails quietly, because the screen always produces something that looks plausible.

That is why the client interface got its own sentence. It is not a synonym for page recognition. It is the case where there is nothing to read and everything to infer.

The shape both products share

One is Gemini on a Mac. One is WorkBuddy, which operates a computer. The descriptions differ. The shape is the same: a model looks at a screen and moves the pointer.

If the shape is the same, the capability is not tied to hardware. Google says its version lives in the model. Tencent says its version does not require the other side's cooperation. Both sentences put the threshold on the agent's own side, which is why both arrived at the same surface.

The side that is hardest to document is that same side.

What is not written down

In the interface era, the boundary lived in the API documentation: what can be called, how often, what an error returns. Those terms were imperfect, but written in advance and auditable.

The screen era has no equivalent yet. Does one click count as one call? Do fifty clicks count as an escalation? Is there a log of what was opened? Three questions, no public answers.

A click also leaves no record by default. An API call tends to leave a timestamp, parameters and a caller. Screen operations produce no such trail unless the product adds one around them, and where that record lives, how long it is kept and who can read it are three more questions.

None of this is a limit of the model. Recognizing a screen and hitting the right pixel means the ability question is settled. The rest is a product decision about how much to delegate, and a system decision about whether to keep the receipt.

The hand exists. The permission slip is still blank.

*Disclaimer: This article is based on publicly available information and is for reference only. It does not constitute investment advice.*

Related coverage

Cite this article

NeuroAI (脑机网), "The Agent Can Click Now. The Permission Boundary Isn't Drawn," 22 November 2026, https://neuroai.site/a/na-model-gemini-computer-use-1122

Machine-readable: BibTeX · RIS · JSON · Markdown

More in “Foundation Models” → · Back to home · Markdown version

China AI, reported clearly
1,300+ verified stories on models, robotics, BCI and the business of AI.
Browse stories