The Wizard of Oz Validation Method
Validate a feature's value and conversion rate before building anything by faking the backend manually.
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 4
- Confidence
- 95%
Before committing engineering resources to a new feature, simulate the entire experience manually so users interact with something that looks real while humans handle the mechanics behind the scenes. This produces real conversion and value-prop data at near-zero cost. Widjaja used this repeatedly at Gojek to test subscriptions, onboarding screens, and personality features without shipping code.
Origin
The 'Wizard of Oz' testing concept is a known lean/UX prototyping technique; Widjaja operationalized it at Gojek at scale, combining it with 'do things that don't scale' (popularized by Paul Graham / Y Combinator).
Core principles
- 01The best possible data is real user behavior in response to a real-feeling experience, not opinion
- 02Every idea is cheap to test at small scale — fake the feature before you fund it
- 03Manifest the intended user experience as fast as possible, then observe conversion
- 04Humans can stand in for backend systems long enough to prove or kill a hypothesis
How to run it
- 1
Define the value prop and metric you want to prove
Decide exactly what conversion or value-prop signal the feature is supposed to produce (e.g. willingness to pay for a subscription package), so the fake test has a clear success reading.
Pro tip Pick a metric you'd use to greenlight the real build, so a pass is directly actionable.
- 2
Design the front-facing experience only
Create just the surface the user sees — a screenshot with an overlaid mockup, an in-app message, a Typeform survey, or a scripted human pitch — nothing on the backend.
Pro tip A designer overlaying a proposed flow on a screenshot of the current screen, sent as an in-app message, is enough to test an onboarding change.
- 3
Wire humans into the backend
Have people manually perform what software would eventually automate — looking up customers, issuing vouchers, deducting balances — coordinated over channels you already control like a WhatsApp group of drivers plus interns 'sitting by the phone.'
Pro tip Use assets you already have (driver phone numbers, voucher distribution, back-end DB access) so setup takes hours, not weeks.
Watch out This only works at limited volume; it's a validation tool, not a launch. Don't confuse a manual pilot with a scalable system.
- 4
Read the conversion signal and decide
Compare the observed conversion/value-prop rates against your threshold. If it clears the bar, build it for real; if not, you've saved the entire engineering cost.
In the wild
To test a subscription feature, the team added ~100 random drivers to a WhatsApp group, told each they were the 'only driver' who could sell a subscription, and had them pitch riders for $10 cash. When a rider said yes, an intern by the phone looked up that customer in the backend, manually granted the vouchers, and deducted $10 from the driver's balance — no product was built.
→ Validated the subscription value prop and expected conversion rates without any engineering work, years before subscriptions officially launched in the market.
Rather than wait on a large engineering effort, the team took a screenshot of the existing screen, had a designer overlay what the new onboarding flow might look like, and pushed it as an in-app message.
→ Got a read on the proposed onboarding flow's reception before building it.
Common mistakes
Building the feature to 'see if people want it'
Founders grasp at new features without any Wizard of Oz test or data that users want the behavior. If you can't devise a way to fake and test it, the idea is effectively useless — build only after a manual test shows demand.
Treating the manual hack as the product
The manual backend proves value but cannot scale; teams err by shipping the hacky version to everyone instead of using the validated signal to justify a real, scalable build.
Is it for you?
Best for
Early-stage founders and growth PMs who want to validate a new feature's conversion before allocating scarce engineering time.
Not ideal for
Features that are already validated and simply need scaling, or regulated flows where a manual backend would create compliance risk.
From the transcript
“it's really this wizard of oz experience we don't have to build anything i coordinated with a bunch of interns and we were able to…”
“when we wanted to do a new onboarding screen but turns out we had lots of engineering work to do we took a screenshot of…”
“if you don't have a tested hypothesis if you can't think of a way to run an experiment then honestly that idea is pretty useless”
From the episode
How to scrappily hire for, measure, and unlock growth
Crystal Widjaja, Gojek and Kumu