The Economic Turing Test
Measure transformative AI by whether you'd hire an agent for a real job without knowing it's a machine
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 90%
Instead of arguing about whether AI is 'AGI', measure whether it can actually do money-weighted jobs. Contract an agent for a job for one to three months; if you would choose to hire it and only then discover it was a machine, it passed the economic Turing test for that role. Aggregate across a market basket of jobs to get an objective transformation threshold.
Origin
Ben Mann explicitly disclaims authorship ('I didn't come up with this, but I really like it'). The economic Turing test framing is associated with AI economists and Mustafa Suleyman's 'modern Turing test'; Mann adapts it into a money-weighted basket-of-jobs metric analogous to how CPI uses a market basket of goods.
Core principles
- 01Judge AI by objective economic transformation, not by the loaded, unfalsifiable term 'AGI'
- 02A job is a concrete, contractable unit you can actually test against
- 03Weight jobs by money, not by count, so the metric tracks real economic impact
- 04Exact thresholds don't matter; the crossing point signals the start of a new era
How to run it
- 1
Pick a specific role and contract an agent for it
Choose one real job and hire an AI agent to perform it over a meaningful window (one to three months) rather than a toy demo.
Pro tip Use a real economic engagement with real stakes, not a benchmark, so you capture the messy long-horizon parts of the job.
- 2
Decide whether you'd actually keep it
At the end of the contract, ask whether you would choose to hire this worker on the merits of the output alone.
Watch out Blind yourself to whether it's human or machine while judging, or the test collapses into bias.
- 3
Reveal and score
If you'd hire it and it turns out to be a machine, it passed the economic Turing test for that role.
- 4
Aggregate into a money-weighted basket
Repeat across many roles. When agents pass for roughly 50% of money-weighted jobs, you have transformative AI and should expect large GDP and societal effects.
Pro tip Track this as a portfolio metric over time to see the knee of the curve coming rather than being surprised by it.
Watch out Societal institutions are sticky and slow to change, so passing the threshold precedes visible disruption; don't wait for the disruption to believe the metric.
In the wild
Mann cites Intercom's Finn resolving 82% of customer service tickets automatically without a human, and Claude Code writing ~95% of Anthropic's own code, as early instances of agents doing money-weighted work end-to-end.
→ Partial role automation today, with humans redeployed to the harder residual tasks rather than eliminated wholesale.
Common mistakes
Arguing about the word 'AGI' instead of measuring impact
'AGI is kind of a loaded term'; debating whether a model can 'do literally everything' is unfalsifiable, whereas the economic Turing test gives an objective, contractable yardstick.
Counting jobs instead of weighting by money
Automating many low-value tasks looks impressive but understates or overstates real transformation; weighting by money keeps the metric tied to economic reality.
Is it for you?
Best for
Founders, investors, and policymakers who need an objective, non-hype way to judge how close AI is to reshaping the economy
Not ideal for
Assessing AI on non-economic dimensions like scientific creativity, companionship, or safety, which the test deliberately ignores
From the transcript
“if you contract an agent for a month or three months on a particular job, if you decide to hire that agent and it turns…”
“if the agent can pass economic turning tests for like 50% of money weighted jobs, then we have transformative AI”
“I think AGI is kind of a loaded term and so uh I tend not to use it very much anymore”
From the episode
Anthropic co-founder on quitting OpenAI, AGI predictions, $100M talent wars, 20% unemployment, and the nightmare scenarios keeping him up at night
Ben Mann