Designing Interfaces an AI Agent Can Use
An agent reaches your product through the screen, the HTML or an API. How to design each layer: accessible names, stable actions and errors it can understand.
This article focuses on the interface and the HTML. For the general framework (what agent experience is and how to evaluate it), see agents as users.
Why can an agent fail where a person does not?
A person infers a lot from the screen: they know a “Continue” button at the bottom right moves the flow forward. An agent has to get that information from somewhere else, and where depends on the tool it uses. Research on computer-use agents shows the problem is rarely the click itself. The OSWorld 2.0 benchmark (June 2026), with long tasks and mid-task changes, concludes that basic interface control is not the bottleneck: agents fail at holding constraints, updating when new information arrives and verifying their result. Your design cannot fix the model’s reasoning, but it can make verification and recovery easier.
Through which route does an agent reach your product?
| Route | What the agent sees | What to design |
|---|---|---|
| Screen (computer use) | Screenshots; clicks at coordinates | Large targets, visible state, no unexpected layout shifts |
| HTML and accessibility tree | Roles and names of elements | Semantic HTML and accessible names |
| API or MCP server | Actions with parameters and results | Stable names, coded errors, verifiable results |
One product can receive all three. Review each separately.
What changes if the agent looks at the screen?
With computer use, Claude controls a desktop through screenshots and mouse and keyboard actions, and coordinates are pixels in the screenshot. Anthropic’s documentation lists limits that translate into design decisions: dropdowns and scrollbars can be hard to manipulate with the mouse, the agent may assume an action worked without checking, and every action is a round trip with latency. For your interface, that means:
- Large, well-separated targets, not small icons packed together.
- Visible state after each action: a confirmation message or a clear change on the page, so the next screenshot allows verification.
- A stable layout: OSWorld 2.0 documents cases where, in interfaces that update in real time, actions target screen positions that are no longer valid.
- A keyboard alternative for components that are hard to use with the mouse.
What changes if the agent reads the HTML?
Tools such as Playwright MCP give the model structured snapshots of the accessibility tree instead of images, so they need no vision model and avoid the ambiguity of screenshots. For that agent, what exists is what the tree exposes: each element’s role and name. Your accessibility work becomes work for agents:
- Use native HTML: a
<button>and a<label>already expose role and name. The WAI-ARIA guidance on names and descriptions says to rely on HTML naming techniques whenever possible. - Make the name say what the control does. Two buttons labeled “Edit” with no context are indistinguishable; “Edit shipping address” is not.
- Do not repeat the role in the name: the WAI-ARIA guidance prohibits it (“button” in a button’s name).
- Be careful with
aria-labelon buttons and links: it replaces the visible text that would otherwise provide the name, and can leave the agent (and a screen reader) with a label different from what the person sees.
A practical consequence: if a screen reader cannot use your flow, an agent that reads the accessibility tree probably cannot either. It is a cheap test.
What changes if the agent calls an API?
When there is an action layer (an API or an MCP server), the agent does not depend on buttons. Four decisions:
- Stable names. “cancel_subscription” keeps its name even if the button text changes.
- Clear parameters. What data it needs, which fields are required, and what format they expect.
- Verifiable result. The action reports whether it completed, what changed, and how to check it.
- Explicit permission. What the agent may do, within what scope, and when it must ask for confirmation.
An error written for a person says “Something went wrong”. An agent needs a stable code, the cause in one sentence and the next possible step: “PAYMENT_DECLINED: the payment method expired; the user must update it”. Without a code, the agent may retry blindly, and that can repeat a charge or a send. If an action can fail in five ways, all five need a code and an exit.
Document each action in a few lines: what it does, which parameters it takes, what it returns, which errors it returns and whether it requires confirmation. If you cannot describe its result in one sentence, it probably does too much. For the MCP server itself, see MCP explained for designers.
What should a person confirm?
Not everything: asking for confirmation at every step makes the agent useless, and never asking makes it dangerous. Anthropic’s computer use documentation recommends requiring human confirmation for consequential actions, such as financial transactions or accepting terms. A practical rule:
- Actions that are hard to reverse: deleting, sending, publishing.
- Money or personal data: charges, changes to data.
- Outside the authorized scope: any step the user did not approve in advance.
Everything else can run with a log of what it did. For designing those screens, see approving an agent’s actions.
Steps to test your interface with an agent
- Choose a real task in your product, with a clear start and end.
- Have an agent attempt it through each route you offer: screenshots only, accessibility tree only and, if it exists, the API only.
- Log where it gets stuck, what confirmation it asks for and what information it lacks.
- Fix the name, permission or state that caused the stall.
- Repeat until the task ends with the correct result and a clear log.
Example
Illustrative example: a booking application has a four-screen flow for rescheduling an appointment. An agent using the accessibility tree stops on the third screen: there are three buttons called “Choose” with no context, and it cannot tell which one matches the desired time. The team renames the buttons (“Choose Tuesday 14, 10:00”), shows a confirmation message after booking and adds a “reschedule_appointment” action with date and time as parameters. The person confirms only when the change involves a charge.
Common mistakes
- Designing actions only as buttons, with no name or parameters.
- Using several controls with the same accessible name.
- Covering visible text with an
aria-labelthat says something else. - Requiring confirmation at every step until the agent becomes useless.
- Returning vague results such as “done” without saying what changed.
- Testing only the happy path and not errors or changes in the middle of a task.
When NOT to expose the interface to agents
When an action has serious consequences and cannot be reversed, or when the product has no records to audit what an agent did. In those cases, design the audit trail and human confirmation first. See also computer use for the risks of letting an agent control a screen.