Itamar Gilad (Gmail, YouTube, Microsoft)21 September 2023

Becoming evidence-guided

6Frameworks

14Insights

Episode overview

Lenny Rachitsky speaks with product coach and former Google product manager Itamar Gilad about replacing opinion-based product development with evidence-guided decisions. Gilad contrasts Google+ with Gmail’s successful tabbed inbox and explains his GIST model: goals, ideas, steps, and tasks. They explore metrics trees, ICE prioritization, the Confidence Meter, inexpensive validation tests, outcome roadmaps, and the GIST Board as practical ways to learn before making larger investments.

Key ideas

Evidence-guided product development balances human judgment with evidence instead of treating opinions or data as sufficient by themselves.
Clear goals should connect value delivered to users, represented by a North Star metric, with value captured by the business through a top KPI.
Metrics trees expose the measurable drivers of company outcomes and help align teams, ownership, prioritization, and organizational structure.
ICE evaluates ideas through impact, confidence, and ease, while the Confidence Meter distinguishes weak opinion-based support from stronger behavioral evidence.
Teams can reduce risk by progressing from inexpensive assessment and data gathering to fake tests, prototypes, experiments, and staged releases.
The GIST Board connects a small set of goals to prioritized ideas and learning-focused steps, giving delivery teams more context and discovery ownership.
Outcome roadmaps preserve flexibility by committing to desired results rather than prematurely committing to low-confidence feature releases.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Productivity3 steps

ICE Prioritization (Impact, Confidence, Ease)

Score each idea on Impact, Confidence, and Ease to bubble the best ideas up without over-engineering the math

Read →

Strategy3 steps

Metrics Trees

Decompose your North Star metric and top business KPI into a tree of sub-metrics to align teams and estimate impact

Read →

Innovation3 steps

The Confidence Meter

Score how much evidence actually backs an idea, from a shiny pitch deck (0.1) to a live A/B test (10)

Read →

Leadership4 steps

The GIST Board and Outcome Roadmaps

Replace release roadmaps with a living board of goals, ideas, and validation steps that the whole team owns

Read →

Innovation4 steps

The GIST Model

Split product work into Goals, Ideas, Steps, and Tasks so evidence — not opinion — drives what gets built

Read →

Innovation4 steps

The Validation Gamut (Assessment, Data, Tests, Experiments, Release)

Validate an idea's assumptions cheaply first — assessment and data before you ever build, fakes before you build for real

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 9

Itamar GiladMentionsItamar is a product coach author speaker and former long time product manager at Google where you worked on Gmail identity and YouTube
“today my guest is idamar galad”
▶ Watch at ~0:30approx.
Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter
“welcome to Lenny's podcast where interview world-class product leaders and growth experts”
▶ Watch at ~0:30approx.
Steve JobsMentionsSteve Jobs is just uh the bar he held for the company and for technical talent and for excellence was not uh wavering.
“it's not the real story at all now we know whatever actually happened and the iPhone is actually a story of discovery”
▶ Watch at ~15:30approx.
Larry PageMentionswhen I was at Google the thing I took away most from any review and Lariat survey
“could you have come to Larry and Sergey and be like here's all this data I've gathered”
▶ Watch at ~16:30approx.
Sergey BrinMentionswhen I was at Google the thing I took away most from any review and Lariat survey
“people are not shy to tell even Sergey and Larry that they are wrong”
▶ Watch at ~17:00approx.
Sean EllisMentionsAssociated with the product-market-fit survey question about disappointment if a product disappeared.
“he's like one of the photos of the growth movement the coined the term growth hacking”
▶ Watch at ~36:00approx.
Sean EllisCoinedAssociated with the product-market-fit survey question about disappointment if a product disappeared.
“Sean invented interesting ice”
▶ Watch at ~36:00approx.
David J. BlandCoinedmentioned
“you do assumption mapping which is a great tool by David J Bland”
▶ Watch at ~51:00approx.
Marty CaganCoinedMarty has been helping product teams and product managers improve their craft processes and careers for over 20 years
“this is MultiCare again and his colleagues they write some tremendous books”
▶ Watch at ~1:07:00approx.

Resources · 37

YouTubeMentionswebsite · Google
“former long time product manager at Google where you worked on Gmail identity and YouTube”
▶ Watch at ~0:30approx.
Lenny's PodcastCoinedpodcast · Lenny Rachitsky
“welcome to Lenny's podcast where interview world-class product leaders and growth experts”
▶ Watch at ~0:30approx.
Evidence-Guided: Creating High Impact Products in the Face of UncertaintyCoinedbook · Itamar Gilad
“he also just published an awesome new book called evidence guided creating high impact products in the face of uncertainty”
▶ Watch at ~1:00approx.
EzraUsescompany
“I actually used Ezra earlier this year unrelated to this podcast completely on my own dime”
▶ Watch at ~1:30approx.
VantaMentionssoftware · Christina Cacioppo
“this episode is brought to you by vanta helping you streamline your security compliance to accelerate your growth”
▶ Watch at ~3:30approx.
GmailUsessoftware · Google
“I joined Gmail in August 2011”
▶ Watch at ~4:30approx.
GoogleUsescompany · Google
“Google Plus was my first experience at gmail I joined Gmail in August 2011”
▶ Watch at ~4:30approx.
GoogleUsescompany · Google
“the solution the obvious solution was to launch the social network of Google called Google Plus”
▶ Watch at ~5:00approx.
FacebookMentionscompany · Meta
“Facebook was massive it's still massive but then it was growing like mushrooms”
▶ Watch at ~5:00approx.
SnapchatMentionssoftware · Snap Inc.
“Google missed the opportunity of social mobile apps like WhatsApp like Snapchat”
▶ Watch at ~6:30approx.
WhatsAppMentionssoftware · Meta
“Google missed the opportunity of social mobile apps like WhatsApp like Snapchat”
▶ Watch at ~6:30approx.
Tabbed InboxCoinedproduct · Google
“the next project after Google Plus was the tabbed inbox”
▶ Watch at ~10:00approx.
AppleMentionscompany · Steve Jobs, Steve Wozniak, and Ronald Wayne
“people showed him all sorts of evidence that Apple can make a phone”
▶ Watch at ~19:00approx.
GISTCoinedother · Itamar Gilad
“this is the gist model goals are there steps and tasks”
▶ Watch at ~21:30approx.
Lean StartupMentionsbook · Eric Ries
“it's based on Lean Startup on design thinking product Discovery growth”
▶ Watch at ~22:00approx.
JiraMentionssoftware · Atlassian
“tasks are the things we manage you know in kanban and jira and all these good tools”
▶ Watch at ~22:00approx.
AirbnbMentionscompany · Brian Chesky, Joe Gebbia, and Nathan Blecharczyk
“in Airbnb I think one of your key metrics or the real North Star metric was Knights booked”
▶ Watch at ~27:30approx.
AmplitudeMentionssoftware · Amplitude
“in amplitude for example they measure Active Learning users or weekly Active Learning users”
▶ Watch at ~28:00approx.
ICE scoring modelUsestool · Sean Ellis
“in the book I suggest using ice impact confidence and ease”
▶ Watch at ~34:30approx.
ICE scoring modelCoinedtool · Sean Ellis
“I'm using the the definitions the of Sean Alice Sean invented interesting ice”
▶ Watch at ~36:00approx.
Confidence MeterCoinedtool · Itamar Gilad
“for that I created a tool called the the confidence meter”
▶ Watch at ~38:00approx.
Assumption MappingRecommendstool · David J. Bland
“you do assumption mapping which is a great tool by David J Bland”
▶ Watch at ~51:00approx.
MicrosoftUsescompany · Bill Gates and Paul Allen
“when I joined Microsoft many years ago the first thing I noticed was that Outlook was very buggy”
▶ Watch at ~54:00approx.
OutlookUsessoftware · Microsoft
“when I joined Microsoft many years ago the first thing I noticed was that Outlook was very buggy”
▶ Watch at ~54:00approx.
GIST BoardCoinedtool · Itamar Gilad
“one of the tools I suggest again this is a process is what I call the gist board”
▶ Watch at ~57:00approx.
AsanaMentionssoftware · Asana, Inc.
“how does this relate to a road map that they may have in a spreadsheet or in jira or an Asana”
▶ Watch at ~1:02:30approx.
Evidence-Guided: Creating High Impact Products in the Face of UncertaintyRecommendsbook · Itamar Gilad
“I try to condense much of what we just discussed and much more a lot more Nuance in the book”
▶ Watch at ~1:04:30approx.
EvidenceGuided.comRecommendswebsite · Itamar Gilad
“there's also the books Landings page where you'll find everything you need to know about the book evidenceguided.com”
▶ Watch at ~1:06:30approx.
EmpoweredRecommendsbook · Marty Cagan and Chris Jones
“one is the series published by s3pg Silicon Valley Pro product group so inspired empowered now I think transformed”
▶ Watch at ~1:07:00approx.
InspiredRecommendsbook · Marty Cagan
“one is the series published by s3pg Silicon Valley Pro product group so inspired empowered now I think transformed”
▶ Watch at ~1:07:00approx.
TransformedRecommendsbook · Marty Cagan, Chris Jones, Christian Idiodi and Jon Moore
“inspired empowered now I think transformed this has come out I haven't read it yet but I'm sure it's amazing”
▶ Watch at ~1:07:00approx.
Lean EnterpriseRecommendsbook · Jez Humble, Joanne Molesky, and Barry O'Reilly
“The Lynn Stout Upland Enterprise lean analytics there's gold in all these books”
▶ Watch at ~1:07:30approx.
Running LeanRecommendsbook · Ash Maurya
“a running Lane that's another example”
▶ Watch at ~1:07:30approx.
Lean AnalyticsRecommendsbook · Alistair Croll and Benjamin Yoskovitz
“Enterprise lean analytics there's gold in all these books Linux really really important books”
▶ Watch at ~1:07:30approx.
Dreaming SpanishRecommendswebsite · Pablo Roman
“I discovered this channel called Dreaming Spanish which is if you're learning Spanish it's incredible so that's my recommendation”
▶ Watch at ~1:08:00approx.
HaifaMentionsplace
“if you go visit Haifa which is the city where I grew up definitely check up the Shawarma”
▶ Watch at ~1:11:30approx.
Lenny's PodcastMentionspodcast · Lenny Rachitsky
“you can find all past episodes or learn more about the show at lennyspodcast.com”
▶ Watch at ~1:12:30approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 3

Myth Buster15:30

Steve Jobs Didn't Dream Up the iPhone in His Kitchen

Itamar pushes back on the popular story that Steve Jobs simply brainstormed the iPhone and told the team to build it. The real history is one of discovery, trial and error, and multiple failed multi-touch and phone projects. Jobs was actually against making a phone for a while; he was the architect who connected the dots as evidence and demos accumulated, not the lone creator of a fully-formed brainchild.

The "Jobs brainstormed it and told the team to build it" narrative is the story he told, not what happened.
The iPhone came out of trial and error and multiple mostly-failed multi-touch and phone projects.
Jobs opposed making a phone and opposed multi-touch initially, then changed his mind when shown evidence.
Founders' instincts matter, but the job is to look at them critically and ask "where's your evidence?" rather than shut them down.

“that's the story Steve Jobs told but it's not the real story at all”
Itamar Gilad · 15:30

“it wasn't his brainchild it was actually against it for a while”
Itamar Gilad · 16:00

#product#apple#founders#evidence-guided

Myth Buster26:30

Your North Star Metric Isn't Your Most Important Metric

Itamar argues most companies mislabel their North Star — what they call it is really just their single most important metric. A true North Star measures how much value you deliver to the market, distinct from the top business KPI (revenue/profit) that measures value captured. WhatsApp's messages sent, Airbnb's nights booked, and Amplitude's weekly active learning users are examples of value-delivered metrics.

The North Star should measure value delivered to users, not the number a company cares about most.
Pair it with a top business KPI (revenue or profit) that measures value captured — the two sides of a value exchange.
WhatsApp measured messages sent; each message is incremental value versus SMS.
Airbnb's real North Star was nights booked; Amplitude's is weekly active learning users who share an insight with two-plus others.

“the northern metric measures how much value we create for the market for example let's take WhatsApp WhatsApp for very long time measured messages sent”
Itamar Gilad · 27:00

#metrics#north-star#product#growth

Myth Buster37:30

How People Quietly Break ICE Prioritization

Itamar explains that ICE (impact, confidence, ease, originally coined by Sean Ellis with "ease" not "effort") gets subverted when people rate impact on gut instinct and then hand themselves a high confidence score to match. Inflating confidence around a guess defeats the whole point of the system, which is why he built the confidence meter to tie a confidence number to the actual strength of evidence behind an estimate.

ICE was coined by Sean Ellis, the growth-hacking pioneer; the original uses "ease" rather than "effort."
People guess impact on instinct, then rate their own confidence high because they feel convinced.
Inflated confidence subverts the whole prioritization system.
The fix is grounding confidence in evidence strength — weak opinion evidence should score near zero.

“people tend to just go with God's Instinct and then give themselves a high confidence they say it's an eight and I'm pretty convinced so…”
Itamar Gilad · 38:00

#prioritization#ice#confidence#product

Hot Take· 2

Hot Take41:00

Copying a Competitor's Feature Is Not Validation

Itamar calls out a common shortcut: if the leading competitor ships a feature, teams treat it as validated and rush to build it. He warns this never works, because you shouldn't assume a competitor understands their users any better than you understand yours. Anecdotal signals like one competitor's move are low-confidence evidence, not proof.

"The competitor has it, so it's a great idea" is treated as validation in many companies.
Copying on that basis never works, in Itamar's experience.
You cannot assume a competitor knows what they're doing any more than you do.
A single competitor's feature is anecdotal, low-confidence data.

“validation is done let's launch it that's it it's a great idea we need to do it never works honestly”
Itamar Gilad · 41:30

“you should not assume that your competitor actually knows what they're doing anymore than you do”
Itamar Gilad · 41:30

#product#competitors#validation#hot-take

Hot Take44:30

Measure Time-to-Outcomes, Not Time-to-Production

Itamar reframes the classic build-fast-versus-learn tradeoff as a wrong metric. Under uncertainty, speed of shipping bits to production is the wrong thing to optimize — what matters is getting the right bits to production and creating the outcomes you need. He argues the evidence-guided method is actually faster and more resource-efficient than opinion-based building, which wastes effort on the wrong things and learns too late.

The right metric is time-to-outcomes, not time-to-production.
Under uncertainty, shipping fast to production optimizes the wrong thing.
Opinion-based methods waste resources building the wrong things and learning too late.
It's a fallacy that learning means building slowly — good teams build and learn at the same time.

“it's not about getting the bits to production it's about getting the right bits to production it's about creating the outcomes that you need”
Itamar Gilad · 45:00

#product#outcomes#speed#hot-take

Explainer· 1

Explainer19:30

Telltale Signs You're Not Actually Evidence-Guided

Many teams assume they run on data but don't. Itamar lists the warning signs: goals that are unclear, too many, vague, or output-focused; missing user-facing metrics (only revenue/business numbers); heavy time sunk into planning and perfect roadmaps; little real experimentation or learning; and disengaged engineers who feel measured only on shipping output.

Goals are unclear, plentiful, vague, or framed as output.
Metrics are missing or purely business-side (revenue) with no user-facing measures.
Lots of time goes into road-mapping, but little experimentation — and where there is, little learning.
Engineers are disengaged because they're signaled to just deliver output, not care about users or the business.

“the goals are very unclear either there are many or they're very kind of obscure and vague or they're about output”
Itamar Gilad · 19:30

“the engineers are kind of getting the signal that what they need to do is deliver they're focused on output”
Itamar Gilad · 20:30

#product#metrics#diagnostics#evidence-guided

Story· 3

Story04:30

Google Plus: The Opinion-Based Bet That Missed WhatsApp

Itamar describes joining Gmail in 2011 to wire it into Google Plus, a social-network project Google threw a ~1,000-person division at to counter Facebook. Everyone believed in it, all the early signals looked good, and none of it worked — users didn't want another social network. Worse, while Google poured millions of person-hours in, it missed the far easier social-mobile opportunities like WhatsApp and Snapchat. For Itamar this became the epitome of what he calls opinion-based development.

Google created a whole new division around Google Plus, roughly a thousand people at its peak — the size of Android or Docs.
Despite early positive indications, people didn't need or use another social network; Gmail later rolled back all the integration and Google Plus shut down in 2018.
Google not only bet on the wrong thing, it missed easier wins — WhatsApp and Snapchat were the real threat to Facebook.
Itamar admits he was often the PM pushing the big idea; the failure became his turning point toward evidence-guided thinking.

“none of it worked turned out people actually didn't need another social network people didn't love it people didn't use it”
Itamar Gilad · 06:00

“at its peak it was about a thousand people inside”
Itamar Gilad · 07:30

#product#google#opinion-based#strategy

Story10:00

The Tabbed Inbox: The Idea No One Believed In

The project right after Google Plus was the reverse of it — a tiny idea nobody believed in. Instead of pushing his own plan, Itamar's colleagues pointed out that similar inbox-organizing ideas had already flopped, which sent the team into real user research on the passive users drowning in clutter. Rigorous testing produced the promotions/social tabs, a feature that made no sense to power users but that ~85–88% of the population loved. Gmail now has ~1.8 billion active users, most using the feature.

It started as a very small idea nobody believed in, the opposite of the top-down Google Plus.
Most colleagues manage their own inbox, so splitting promotions and social sounded like the stupidest idea to them.
About 85–88% of users — the passive, cluttered majority — absolutely love it.
The difference from Google Plus: the team didn't have high confidence in their opinions, so they used evidence instead of going all in.

“it started as a very small idea that no one believed in”
Itamar Gilad · 10:00

“there's about 85 percent of the population 85 to 88 that absolutely love it and today Gmail has about 1.8 billion active users”
Itamar Gilad · 11:30

#product#gmail#user-research#evidence-guided

Story52:00

Faking Gmail's Tabbed Inbox With Zero Code

To validate the tabbed inbox cheaply, the team ran a Wizard of Oz test: an early version shown to users wasn't really Gmail but a facade of HTML. With user permission, team members manually sorted just the subject and sender of the top 50 messages into the right place while an interviewer distracted the person, then revealed a tidy inbox. Users said "wow, this is very cool" — evidence gathered without a single line of code written.

The demo was an HTML facade, not real Gmail.
Team members hand-sorted the top ~50 messages' subject and sender behind the scenes, with permission.
An interviewer distracted users before revealing the sorted inbox.
The positive reactions gave enough evidence to justify building it — with no code written.

“it wasn't really Gmail it was just a facade of HTML and behind the scenes and according to the permissions that the users gave us…”
Itamar Gilad · 52:30

“there wasn't a single line of code that's written”
Itamar Gilad · 53:00

#product#testing#wizard-of-oz#gmail

Q&A· 1

Q&A18:30

How to Challenge a Boss's Idea With Evidence

Asked how someone at a less-open company can push back on a founder or exec, Itamar's tactical answer is to run a quiet experiment and bring back real data. One of two things happens: they get furious and tell you to get back to work — in which case that person is unreasonable and you should start job-hunting — or, far more commonly, they are pleasantly surprised and change their mind, exactly as Steve Jobs did when shown Apple could build a phone.

Run a secret experiment and come back with hard data rather than arguing opinion-to-opinion.
Outcome one: they get mad and dismiss you — a sign the person is unreasonable and you may need to look elsewhere.
Outcome two (more common): they are pleasantly surprised and flip their view.
In opinion-based debates the most senior person's opinion usually wins; evidence empowers junior and mid-level people to challenge that.

“either they get extremely mad at you and they tell you to get back to work and to do what you were told and in…”
Itamar Gilad · 18:30

#product#leadership#influence#evidence-guided

Tool· 1

Tool1:08:30

ElevenLabs for Cloning Your Own Voice

Itamar's recently-discovered favorite product is ElevenLabs, an AI voice company. Beyond having the best synthetic voices he's heard, it can replicate your own voice into a reusable signature — useful for narrating an audiobook or an online course. He jokes it's part of his "retirement plan" to assemble AI components that can eventually stand in for him.

ElevenLabs produces the best synthetic voices he's heard.
It can clone your own voice into a personal voice signature.
Practical uses: narrating an audiobook or online course.
Framed half-jokingly as building an AI version of himself.

“there's a company called 11 Labs uh that do voices and like the best voices synthetic voices you heard but they can also replicate your…”
Itamar Gilad · 1:09:00

#tools#ai#voice#elevenlabs

Takeaway· 3

Takeaway43:00

Don't Test Everything — Know When to Stop

Being evidence-guided doesn't mean testing every change to the hilt. Itamar's counterbalance: match investment to the confidence you need, and recognize when expert opinion alone is enough. Low-risk, low-visibility changes — like reordering settings no one sees — can just be launched. Part of the skill is knowing when to stop climbing the confidence meter.

Tie your investment to the level of confidence you actually need.
Some changes need no test — expert opinion suffices when risk and visibility are low.
Reordering settings nobody notices is a launch-without-testing case.
Knowing when to stop matters as much as knowing how to build confidence.

“some ideas you don't need to test sometimes expert opinion is enough if you're just changing the order of the settings no one sees this…”
Itamar Gilad · 43:00

#product#testing#prioritization#takeaway

Takeaway47:30

The Two Types of Companies This Helps Most

Itamar identifies two audiences that respond most strongly to evidence-guided methods. First, companies emerging into modern product development — they have PMs, OKRs, and Agile and are starting to experiment but can't yet stitch it together. Second, companies that used to be evidence-guided and regressed, often after a change of management or culture, and now need to rekindle a lost spirit.

Type one: companies moving into modern product management, doing bits of experimentation but struggling to unify it.
In those firms every CPO ends up building their own little framework.
Type two: once-evidence-guided companies that regressed, which happens too often.
Regression usually follows a change of management or culture, requiring a deliberate rediscovery.

“the other type is those companies that used to be evidence guided and they regressed and that happens way too often”
Itamar Gilad · 47:30

#product#organizations#transformation#takeaway

Takeaway1:09:30

Strive Not to Be a Success, But to Be of Value

Itamar's guiding motto, attributed to Albert Einstein, is to strive to be of value rather than to be a success. He ties it to his whole value-exchange philosophy for products and companies. Lenny extends it to content creators: chasing followers and success fails, while delivering genuinely valuable, interesting work is the real signal that resonates with others.

The motto: strive not to be a success but to be of value.
It loosely underpins Itamar's value-exchange concept for products.
Applied to content: creators chasing success and followers usually fail.
The signal for value is whether you yourself find it interesting and valuable.

“strive not to be a success but to be of value”
Itamar Gilad · 1:09:30

#mindset#value#motto#takeaway

Key ideas

Frameworks in this episode

ICE Prioritization (Impact, Confidence, Ease)

Metrics Trees

The Confidence Meter

The GIST Board and Outcome Roadmaps

The GIST Model

The Validation Gamut (Assessment, Data, Tests, Experiments, Release)

People & resources mentioned

People · 9

Resources · 37

Insights & moments

⚡Myth Buster· 3

Steve Jobs Didn't Dream Up the iPhone in His Kitchen

Your North Star Metric Isn't Your Most Important Metric

How People Quietly Break ICE Prioritization

◆Hot Take· 2

Copying a Competitor's Feature Is Not Validation

Measure Time-to-Outcomes, Not Time-to-Production

✶Explainer· 1

Telltale Signs You're Not Actually Evidence-Guided

❝Story· 3

Google Plus: The Opinion-Based Bet That Missed WhatsApp

The Tabbed Inbox: The Idea No One Believed In

Faking Gmail's Tabbed Inbox With Zero Code

?Q&A· 1

How to Challenge a Boss's Idea With Evidence

✦Tool· 1

ElevenLabs for Cloning Your Own Voice

▲Takeaway· 3

Don't Test Everything — Know When to Stop

The Two Types of Companies This Helps Most

Strive Not to Be a Success, But to Be of Value

Myth Buster· 3

Hot Take· 2

Explainer· 1

Story· 3

Q&A· 1

Tool· 1

Takeaway· 3