Computer Use: AI That Operates the Screen the Way a Person Does

Computer use is AI that sees the screen, moves the cursor and types to finish tasks in any application. It is useful, but in 2026 it still fails on long tasks.

info Quick Definition
Computer use is the ability of an AI model to observe a computer screen, interpret what it sees and carry out mouse and keyboard actions to complete a task in any application. It operates the same interface a person uses, without needing a specific API.

How does it work?

The loop is simple: the agent receives a screenshot, decides on an action (click a coordinate, type text, scroll), executes it, observes the new state and repeats. It is the same agent loop of thinking, acting and observing, with the screen as the environment.

That flexibility has a cost. Every step requires interpreting an image, and long tasks accumulate many steps. An agent filling a form on an old website has to read labels, detect errors and recover if a pop-up changes the order of the fields.

How well does it work today?

The reference benchmark is OSWorld. Its original version evaluates 369 real-world computer tasks, each with an execution-based evaluation script. According to the project site, humans complete more than 72% of those tasks, while at the 2024 release the best agent reached 12.24%.

In June 2026, OSWorld 2.0 appeared, with 108 long-horizon workflows. According to the paper, a person takes a median of about 1.6 hours per workflow (about two minutes in the original), and the heaviest agent used around 318 tool calls per task, compared with roughly 30 before. The best model completed 20.6% of tasks under strict binary scoring and reached a 54.8% partial score at a 500-step limit. Another model was more token-efficient but plateaued near 13%.

The most useful part of the study is not the number but the diagnosis: agents fail mainly at reasoning and task management. They lose constraints, miss information that arrives midway through, guess instead of asking and skip verification. Basic interface control, such as clicking the right place, is no longer the bottleneck.

What do the providers offer today?

The three big labs expose computer use as an API tool in which your application runs the actions and sends back screenshots:

  • Anthropic: the computer use tool in the Claude API. Earlier versions (computer_20251124, computer_20250124) require a beta header; the current toolset (computer_toolset_20260801) does not, although the SDK classes are still in beta.
  • OpenAI: the computer tool in the Responses API, which replaces the earlier computer-use-preview model.
  • Google: the computer_use tool in the Gemini API, with browser, mobile and desktop environments.

All three documentation sets agree on the essentials: isolate the environment, treat screen content as untrusted and ask for human confirmation before consequential actions. Model names and versions change every few months, so check the official documentation before building on them.

When does it make sense?

Computer use makes sense in three cases:

  • There is no API. An old system or a portal without integration.
  • The flow is repetitive and bounded. Clear steps, no high-risk decisions.
  • Important results get human review. The agent proposes, the person confirms.

If a stable API or integration exists, it is almost always better to use it. It is faster, cheaper and less brittle than watching a screen.

Steps to test a use case

  1. Choose a repetitive task with a clear start, a clear end and a verifiable result.
  2. Run it in an isolated environment, with a test account and no access to real data.
  3. Log where the agent fails, what it confirms and what it skips.
  4. Measure time and the number of corrections, not only whether it finished.
  5. Decide whether the saving justifies the supervision before using it with real data.

Example

Illustrative example: a design agency has to download reports every month from a client portal that has no API. A computer use agent runs the route in an isolated environment with a test account. It works three months out of four. In the month it fails, the portal reordered a menu, and the agent downloaded the wrong report without flagging it. The team adds a check: the agent must read the file name and period and compare them with a list before saving. The task is still supervised, but it no longer depends on luck.

What security measures should you apply?

Three basic measures before any real test. The first is isolation: a virtual machine or a dedicated browser profile, with no saved sessions for personal or company accounts. The second is least privilege: the agent sees only the folders and sites it needs for the task, and nothing else. The third is confirmation at irreversible steps, such as submitting a form with payment data or deleting a file.

There is also a less visible risk: a web page can contain text designed to divert the agent, for example hidden instructions asking it to share data. Treat everything the agent reads on screen as untrusted information, not as instructions. Anthropic also recommends limiting internet access to an allowlist of domains and keeping sensitive data such as login credentials away from the model. For the permissions framework, review sandboxing and agent permissions.

What should you log for each run?

Every run should leave a record that can be reviewed without repeating it. Keep the initial screenshot, the list of actions with their timestamps, the final screenshot and the verification result. If the agent fails, the log shows at which screenshot it went off track, something you cannot see in a simple “task failed”. Those records also help compare versions of the flow when the portal changes.

Keep the logs under a clear retention policy, because screenshots can contain personal data. Deleting them on time is part of security, not a storage detail.

Common mistakes

  • Giving the agent access to real accounts in the first test.
  • Measuring only whether the task finished, without counting silent errors.
  • Using it when a stable API exists.
  • Assuming an agent that did three cases well will do the fourth.
  • Forgetting that a page can contain instructions meant to divert the agent.

When NOT to use it

When the task has serious consequences, such as payments, password changes or messages to customers, with no human confirmation. Also when the flow changes often, because the agent breaks with every redesign. To think about the limits of autonomy, review levels of agent autonomy and sandboxing and agent permissions.

Sources

Frequently asked questions

What is computer use?
A model’s ability to interpret screenshots and act with mouse and keyboard, the way a person would, to complete tasks in applications.
How well does computer use work today?
According to OSWorld 2.0 (June 2026), the best model completed 20.6% of the long tasks, with a 54.8% partial score. The main failures are in reasoning and task management, not basic interface control.
When should you use computer use?
When there is no API for the task, the flow is repetitive, every important result gets human review, and it runs in an isolated environment.
What is the main risk?
The agent acting on real data or accounts with no one supervising it, or a malicious page giving it instructions.