Agents learn to see and feel their way through UI
Browser-use agents such as Hark Handoff suggest a shift from prompt-based AI toward systems that can see, plan and act inside live interfaces.
The next breakthrough in AI may not be a better answer inside a prompt box. It may be the moment an agent can use a computer with its eyes open.
Hark points toward a new class of browser-use agent: a system that spins up its own temporary virtual computer in the cloud, complete with a real desktop, browser, file downloads folder, terminal and operating environment. Instead of only reading web text or calling an API, the agent can look at the rendered screen and act on it. It clicks. It scrolls. It types. It waits for the page to change.
This is why the idea feels important for design. In our earlier article, Beyond the prompt window: AI needs to become sensory, we argued that the prompt box is too narrow a way to collaborate with intelligence. Design work is visual, spatial, iterative and embodied. Browser agents suggest a practical version of that argument: AI is beginning to perceive interfaces as environments, not just documents.
A frontier language model is astonishingly capable, but it still mainly produces language: words, code, structured data or tool calls. A virtual computer agent adds another layer. It gives the model a controlled place to operate, with a live browser and a visual surface it can inspect. The web becomes something closer to a workspace than a source of scraped text.
That matters because modern websites are messy. They have cookie banners, hidden menus, interactive checkout flows, modal windows, JavaScript-rendered states and forms that only reveal the next step after the previous one has been completed. A text-only model can describe what should happen. A browser agent can move through the interface itself.
| Feature | Virtual computer agent | Frontier lab LLM |
|---|---|---|
| Primary output | Physical UI actions: clicks, scrolls, typing and cursor movement | Text tokens, code, summaries or structured JSON |
| Operating environment | A live, sandboxed cloud desktop with browser, files and terminal | A chat interface, API context or model runtime |
| Web interaction | Reads and acts on rendered HTML, pop-ups, JavaScript and cookies | Relies on text extraction, APIs or separate browsing tools |
| State and files | Maintains a task session with downloads, uploads and browser state | Relies on context windows, memory systems or supplied files |
| Risk profile | Wrong clicks, blocked flows, credential/payment sensitivity | Hallucination, prompt injection and bad recommendations |
| Best role | Supervised task execution inside real software | Reasoning, drafting, planning, coding and review |
The useful version is not a fully autonomous agent buying tickets while nobody is watching. That is too brittle. Dates, names, payments and commitments still need human approval. The more useful model is supervised delegation: the agent searches, compares, fills forms and prepares the decision, then stops at the moment of purchase or commitment.
For studios, the shift is not only operational. It changes what an AI interface is. Instead of asking an assistant to explain how to book a train, gather references, check a supplier form or file a receipt, the assistant can enter the interface and do the visible work. That makes agency less abstract. You can see what the system is doing, interrupt it, correct it and approve it.
The See-Plan-Act visual loop
1. See the screen
Every few milliseconds, the system captures a high-resolution screenshot of the virtual computer display. To the vision-language model, that image begins as a dense grid of numerical pixel values representing colour, brightness and position.
2. Map the interface
The model places a spatial coordinate map over the browser window. It is no longer only reading labels; it is locating objects in space.
3. Detect the objects
Using visual recognition, the system segments the page into interactive elements. It draws invisible boundary boxes around buttons, text entry fields, dropdown menus, toggles, checkboxes, navigation items and alerts.
4. Plan the next move
The agent compares the visible state of the interface with the user's goal. It decides whether to click, type, scroll, wait, download a file, open a new tab or ask for confirmation.
5. Calculate the target
This is where the interface stops being language and becomes space. Instead of relying only on a button's visible label, such as Watch on YouTube, the agent calculates where that button physically sits on screen. It might decide: "The watch button is at X: 450, Y: 720."
6. Move the virtual cursor
Once the coordinate is chosen, the AI sends a direct instruction to the virtual machine: move the cursor to that point, then click, type or scroll. The action happens inside the isolated cloud computer, not on the user's personal machine.
7. Read the change
The page responds, the screen changes, and the agent looks again. That closes the loop: see, plan, act, then see again.
Alex SwainExperience Director, Studio TypoWe need visual interfaces like websites to learn and comprehend information. But we now need agents to 'see' these as we do, because we built the internet for interacting with.
This is the real breakthrough. Browser agents do not make web design less important; they make it more important. Before Handoff, there was a seductive argument that UI might become redundant because bots could inspect code, parse structure and extract data directly. But code inspection is not the same as completing a checkout, navigating a booking flow, resolving a modal window or making sense of a designed decision path. As more agents browse, work and transact on the internet, visual interfaces will become shared ground between humans and machines. The practice of web design remains vital because agents need the same thing people need: clear information architecture, visible affordances, understandable states and interfaces they can move through with confidence.