Beyond the prompt window: AI needs to become sensory

With so much euphoria around AI technology and how it will change the world, I am researching what it means to be a designer. What is the future of design in the context of agentic companions, creative automation and new forms of brand experience?

In June, SXSW came to London and I had the pleasure of talking with designers from all backgrounds about the disruption and emerging concepts we are trying to predict and understand. 2026 is the year of agents. OpenClaw dropped in March, followed by Claude, Hermes and others. We can download open-source models locally, plug in an agent and we are off. Our sense of agency is rocket fuelled.

Melissa Jackson Parsey, R/GA global chief strategy officer

If the future looks like humans and agents working together, brands and creatives need to play to the strengths of both.

Installing my agent was not a dream exercise. I spent many late evenings with a chatbot and the Terminal window, experimenting with commands until everything snapped aligned and the system was alive. The process felt like jump-starting a car or wiggling the connection to my solenoid until the engine ignited. I had never worked in the Terminal, but navigating it became very familiar. I liked the rawness of the interface, reading lines of code to check for syntax errors.

The experience was amplified when I read my first greeting from the agent: "Hi Alex, what should you call me and what are we going to work on?" We then established a framework through feeding it my design references and interests, plus my career to date and why we need to work together. I had first been introduced to an AI prompt window by OpenAI in 2022, and although the UI has been enriched since then, I am wondering how human interaction with LLMs is in need of more sensory engagement.

I think the ideas Peter Steinberger wired into OpenClaw to make it feel more human were very convincing, especially when my agent wished me happy birthday. Talking with it in the first few weeks was utterly spellbinding, a complete game changer. It now manages my home life and allows me to explore design ideas at home as if I had a small team. The interaction via text or voice messaging feels limiting though. I can research topics, build software and design more things, but orchestrating all this through an input window is not human. Comparing a conversation with a friend, the process feels like completing a form.

In a recent interview with Jony Ive about his Ferrari Luce project, he said: "That's been my focus over the last 40 years: how does humanity connect to the tools that we make? And it can be one that is joyful and enabling and elevating, or it can be one that is frustrating and irritating and disconnecting and isolating."

An input window is like peeping through a letterbox to see the interior details inside a home. We need more visceral ways to engage with artificial intelligence. There are so many opportunities available to creatives. We need to find ways to pull back the curtains, or break the front door down, to reveal the naked intelligence.

Design work never begins as a complete instruction. It often starts with visual research, contextual, social and iterative processes. Design ideas develop on half-formed intent, comparison, critique and taste, until they start to make sense. We qualify the quality of the design through debate with colleagues until it meets our standards, predetermined by years of experience.

At Bloomberg's Tech conference in early June, ex-OpenAI CTO Mira Murati spoke of her work at Thinking Machines, which is trying to define the next wave of AI products through richer interactive models. Her reference was the multisensory nature human interaction takes: body language, listening while also thinking how to respond, everything at once.

The prompt box made AI accessible, but may also be why so many are not engaging with AI in the broader public. AI experience needs to mature into more visually engaging forms, yet also push our thinking and expectations. Murati's argument is not simply that AI should become more autonomous. It is that intelligence becomes more useful when it can collaborate with humans in real time, in ways we already find effective.

When Jony responded to the launch of Ferrari's first electric car, he made the point that people understand the new by comparing it with what they already know. The Luce car is judged against the combustion Ferrari: engine noise, vibration, silhouette and ritual. However, a new design should be tangible enough that it can earn its own expectations.

Jony Ive

I've been so intrigued by how people react and how they feel faced with something new. And as you know, we know everyone has an opinion, but normally what people do in trying to understand what's new, you try to reference what's familiar.

Even though the car is powered by new technology, the interiors and control dashboard are designed to blend digital and physical world ideas. The steering wheel features switches which embrace the mechanical, the utility of controls. The dashboard changes colour once a driving mode is selected, then fades smoothly back to neutral. This is where the digital and physical support one another.

Design work must become tangible for it to be reviewed before improving. A drawn sketch, a prototype, a printout, a model: these are how we create meaning in our work. AI output feels unintelligent when it is just wrong, yet even with the most refined prompts and visual references it lacks a broader understanding of the real world, which is evident in what it produces. Writing code is not what I am judging here. Yes, agents can build as well as humans. My context is design, and the AI needs to see, hear, touch and perhaps smell to really comprehend and perceive our prompts.

If AI can develop into a multisensory device, it transcends being a tool and becomes a true companion. At Google's I/O conference recently, a talking financial advisor was demoed through a partnership with Citi Bank. The full-screen avatar wearing a suit and asking how it could help does not embody trust to me personally. It is clearly fake, a digital puppet. I would rather engage with the Terminal window.

Once AI starts to see, hear, touch and smell, interaction design stops being screen design. It becomes the design of perception itself: what the system notices, how it responds, and how humans understand what it thinks it has sensed. To get to this stage, AI needs to feel and interact with objects like people can. Industrial product design could be the answer: the real-world references of materials and touch. Maybe with the development of new wearables, we can give agents a sense of multitouch for the real world.