AI Computer Interfaces (ACI)
Give AI agents purpose-built text interfaces, not the human GUIs they were never designed for
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 90%
LLMs are trained on human text but behave like 'alien creatures' — the fastest, cheapest way to let them operate software is not to make them drive human interfaces (screenshots, video, mouse) but to build stripped-down text representations of each tool the agent needs. Masad frames this as a new discipline paralleling Human-Computer Interaction (HCI): AI-Computer Interaction (ACI). Instead of forcing the model through expensive visual computer-use, you expose the shell, the editor, and package installs as text feedback loops tuned for how the model actually reasons.
Origin
Articulated by Amjad Masad describing how Replit's agent was engineered; he references anthropic's computer-use as the expensive visual alternative ACI is designed to avoid, and credits the broader emergence of academic 'AI computer interface' papers.
Core principles
- 01LLMs are alien creatures, not humans — don't assume human interfaces suit them
- 02Visual/computer-use interfaces are expensive (image and video processing); text feedback is cheaper and faster
- 03Each tool an agent uses needs its own interface designed for the model, not repurposed from humans
- 04How much to show the model (e.g. how many files) before it hallucinates is an empirical design variable
- 05It is currently more art than science, but becoming science through experimentation
How to run it
- 1
Inventory the tools the agent must operate
List every capability the agent needs — shell, code editor, package manager, database queries, screenshots — and treat each as a separate interface design problem rather than handing the model a raw human GUI.
Pro tip Structure the agent as just another user of your existing system. Replit reused its multiplayer (real-time collaborative) editor so the agent codes as a second user alongside the human.
- 2
Replace visual interfaces with text representations
For each tool, give the model a text feedback channel rather than pixels. For the shell, feed a text representation of what the shell is doing at set increments; for editing, give an editor tool that returns error feedback as plain text, mirroring what a human sees but simpler.
Watch out Making the model use human interfaces (image/video like computer-use) is expensive and slow — reserve visual channels (e.g. screenshots to verify a page rendered) for verification, not primary control.
- 3
Tune information density empirically
Experiment with how much context each interface exposes — how many files to show, at what increment to stream shell output — because too much causes hallucination. Treat these as parameters to test, not fixed choices.
Watch out There is no settled answer yet; expect to iterate as models change.
- 4
Sit it all on top of the strongest available foundation model
Route each sub-task to the model best at it inside a multi-agent system — Replit uses anthropic's Sonnet for coding, OpenAI models for critique/manager roles, and trains its own embedding model for search.
Pro tip Different models have different 'powers' — build a Society of Models rather than forcing one model to do everything.
In the wild
Rather than screen-scraping a terminal, Replit feeds the agent a text representation of shell activity at increments, and an editor tool that surfaces errors as plain text as the agent writes code. Screenshots are used only so the agent can check that a page renders.
→ The agent built a full-stack feature-request app — database, schema, homepage, admin controls — in roughly 5-10 minutes at an estimated ~15 cents of compute, work Masad estimated would take a decent engineer a few days.
Common mistakes
Forcing agents through human visual interfaces
Using computer-use-style image/video control as the default makes the agent slow and expensive because every frame must be processed; a purpose-built text interface achieves the same control far more cheaply.
Overloading the model's view
Showing too many files or too much raw output pushes the model into hallucination; the right density is an empirical limit you must find per interface, not a 'more context is better' default.
Is it for you?
Best for
Engineers and product teams building agentic AI products who need an autonomous model to reliably operate real tools and services
Not ideal for
Simple single-turn chat features where the model never needs to operate external tools
From the transcript
“there's like almost like a new discipline um called um AI computer interfaces”
“so instead we you know for the shell for example we give it a you know a a sort of a text representation of what…”
“trying to make them use human interfaces like anthropics computer use but those are really expensive and you need to kind of process all this…”
“it's going to be Society of models like products will be made of a lot of different models”
From the episode
Behind the product: Replit
Amjad Masad (co-founder and CEO)