The Validation Gamut (Assessment, Data, Tests, Experiments, Release)
Validate an idea's assumptions cheaply first — assessment and data before you ever build, fakes before you build for real
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 90%
The Steps layer of GIST provides a spectrum of ways to validate an idea's assumptions, ordered from cheap to expensive: Assessment (goal alignment, ICE, assumption mapping, stakeholder chats), Data (analytics, surveys, competitive analysis, user interviews, field research), Tests (fakes like smoke/Wizard-of-Oz/concierge tests, then rough builds like early-adopter programs and betas), Experiments (A/B and multivariate tests with a control), and Release (staged rollouts, percentage launches, holdbacks). The core insight: you don't have to start at the expensive right-hand side — start cheap, park weak ideas fast, and invest more only in ideas that keep generating positive evidence.
Origin
Presented by Itamar Gilad as the Steps layer of GIST, explicitly assembled from tools 'much smarter people invented' — e.g. assumption mapping is credited to David J. Bland, and the fake-it tests (smoke test, Wizard of Oz, concierge) come from lean/startup practice. Illustrated with the Gmail tabbed-inbox smoke test Gilad ran at Google.
Core principles
- 01You can validate at a much lower cost than building an elaborate MVP — most teams don't realize this
- 02Test the assumptions inside an idea, not just the finished idea
- 03Start cheap (assessment, existing data) which lets you park many ideas quickly before building anything
- 04Keep research ongoing rather than starting it only after you have an idea, so you always have data to compare against
- 05Even the release is a chance to learn — staged rollouts and holdbacks keep validating, and rollbacks are learning opportunities
How to run it
- 1
Assessment — near-free checks
Check whether the idea aligns with goals, do quick business modeling, run ICE analysis, do assumption mapping (David J. Bland's tool), and talk one-on-one with stakeholders to surface risks. These require little work but teach a lot about impact and ease.
Pro tip These cheap checks alone let you park many ideas before spending anything on data or building.
- 2
Data — evidence you already have or can gather
Mine your analytics, run surveys, do competitive analysis, conduct user interviews, and observe users in field research. Interviews and field research are expensive, so keep research running continuously rather than starting from scratch per idea.
Pro tip Ongoing research means you already have data to compare a new idea against instead of waiting weeks to gather it.
Watch out User interviews and field research are the pricey end of data-gathering — don't leave them until after the idea exists.
- 3
Tests — fake it, then build rough
Start by faking the product: smoke tests, Wizard-of-Oz tests, concierge tests, usability tests — no real code. Then build rough, unpolished, non-scalable versions good enough for users: early-adopter programs, alphas, longitudinal studies, and 'fish food' (testing on your own team, a more local dogfooding).
Pro tip A Wizard-of-Oz test can be run entirely by researchers and designers with zero engineering — a facade plus manual back-end work behind the scenes.
Watch out Don't jump to building a polished version to test — 'initially you don't build anything, you fake it.'
- 4
Experiments and Release — controlled validation at scale
Run true experiments with a control element (A/B tests, multivariate tests). Then validate even during release via staged releases, percentage launches, and holdbacks; roll back and change things when needed.
Pro tip Reserve the word 'experiment' for tests that actually have a control — it keeps the rigor honest.
Watch out Deciding up front with low confidence that a specific idea 'must be launched' by a date suffocates all of this learning.
In the wild
An early version wasn't real Gmail — 'it was just a facade of HTML.' With users' permission, team members manually moved the subject and sender of the top ~50 messages into the right tabs while an interviewer distracted the user, then revealed the sorted inbox. 'There wasn't a single line of code written.'
→ Users reacted with 'wow this is actually very cool,' giving strong evidence to justify building the real feature.
When Gilad joined Microsoft, Outlook was very buggy; the team explained they were all dogfooding the next unreleased version of Outlook — a common Silicon Valley practice of using rough builds internally to validate them.
→ Illustrates the mid-level 'rough build' test class where an incomplete version is used by real internal users to generate evidence.
Common mistakes
Building an elaborate 'MVP' and only learning after launch
Teams believe they must build a big MVP that is minimal in no way, then discover post-launch — 'basically what we used to call beta 20 years ago with a different name.' Cheap earlier validation would have parked the idea sooner.
Treating a competitor's feature as validation
Many companies decide validation is done the moment a leading competitor ships a feature; it 'never works,' because you can't assume the competitor knows what they're doing any more than you do.
Is it for you?
Best for
Product teams that default to building before validating and want a concrete menu of cheaper ways to test an idea's assumptions before committing engineering
Not ideal for
Low-risk, invisible changes where any structured validation is overkill and expert opinion suffices
From the transcript
“organizations don't know that they can actually learn at a much lower cost they believe they need to build this elaborate MVP which is not…”
“assessment effect finding tests experiments and release results”
“you do assumption mapping which is a great tool by David J Bland”
“but initially you don't build anything you fake it you don't fake the or test you do smoke tests Wizard of Oz tests concierge test…”
“fish food is testing on your own team”
“the key point is you don't have to start at the right hand side which is expensive you can start early on and that leads…”
From the episode
Becoming evidence-guided
Itamar Gilad (Gmail, YouTube, Microsoft)