LLenny's Podcast
← All episodes
Itamar Gilad (Gmail, YouTube, Microsoft)21 September 2023

Becoming evidence-guided

6Frameworks
14Insights

Frameworks in this episode

Productivity3 steps

ICE Prioritization (Impact, Confidence, Ease)

Score each idea on Impact, Confidence, and Ease to bubble the best ideas up without over-engineering the math

Read →
Strategy3 steps

Metrics Trees

Decompose your North Star metric and top business KPI into a tree of sub-metrics to align teams and estimate impact

Read →
Innovation3 steps

The Confidence Meter

Score how much evidence actually backs an idea, from a shiny pitch deck (0.1) to a live A/B test (10)

Read →
Leadership4 steps

The GIST Board and Outcome Roadmaps

Replace release roadmaps with a living board of goals, ideas, and validation steps that the whole team owns

Read →
Innovation4 steps

The GIST Model

Split product work into Goals, Ideas, Steps, and Tasks so evidence — not opinion — drives what gets built

Read →
Innovation4 steps

The Validation Gamut (Assessment, Data, Tests, Experiments, Release)

Validate an idea's assumptions cheaply first — assessment and data before you ever build, fakes before you build for real

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 3

Myth Buster15:30

Steve Jobs Didn't Dream Up the iPhone in His Kitchen

Itamar pushes back on the popular story that Steve Jobs simply brainstormed the iPhone and told the team to build it. The real history is one of discovery, trial and error, and multiple failed multi-touch and phone projects. Jobs was actually against making a phone for a while; he was the architect who connected the dots as evidence and demos accumulated, not the lone creator of a fully-formed brainchild.

  • The "Jobs brainstormed it and told the team to build it" narrative is the story he told, not what happened.
  • The iPhone came out of trial and error and multiple mostly-failed multi-touch and phone projects.
  • Jobs opposed making a phone and opposed multi-touch initially, then changed his mind when shown evidence.
  • Founders' instincts matter, but the job is to look at them critically and ask "where's your evidence?" rather than shut them down.

“that's the story Steve Jobs told but it's not the real story at all”

Itamar Gilad · 15:30

“it wasn't his brainchild it was actually against it for a while”

Itamar Gilad · 16:00
#product#apple#founders#evidence-guided
Myth Buster26:30

Your North Star Metric Isn't Your Most Important Metric

Itamar argues most companies mislabel their North Star — what they call it is really just their single most important metric. A true North Star measures how much value you deliver to the market, distinct from the top business KPI (revenue/profit) that measures value captured. WhatsApp's messages sent, Airbnb's nights booked, and Amplitude's weekly active learning users are examples of value-delivered metrics.

  • The North Star should measure value delivered to users, not the number a company cares about most.
  • Pair it with a top business KPI (revenue or profit) that measures value captured — the two sides of a value exchange.
  • WhatsApp measured messages sent; each message is incremental value versus SMS.
  • Airbnb's real North Star was nights booked; Amplitude's is weekly active learning users who share an insight with two-plus others.

“the northern metric measures how much value we create for the market for example let's take WhatsApp WhatsApp for very long time measured messages sent”

Itamar Gilad · 27:00
#metrics#north-star#product#growth
Myth Buster37:30

How People Quietly Break ICE Prioritization

Itamar explains that ICE (impact, confidence, ease, originally coined by Sean Ellis with "ease" not "effort") gets subverted when people rate impact on gut instinct and then hand themselves a high confidence score to match. Inflating confidence around a guess defeats the whole point of the system, which is why he built the confidence meter to tie a confidence number to the actual strength of evidence behind an estimate.

  • ICE was coined by Sean Ellis, the growth-hacking pioneer; the original uses "ease" rather than "effort."
  • People guess impact on instinct, then rate their own confidence high because they feel convinced.
  • Inflated confidence subverts the whole prioritization system.
  • The fix is grounding confidence in evidence strength — weak opinion evidence should score near zero.

“people tend to just go with God's Instinct and then give themselves a high confidence they say it's an eight and I'm pretty convinced so…”

Itamar Gilad · 38:00
#prioritization#ice#confidence#product

Hot Take· 2

Hot Take41:00

Copying a Competitor's Feature Is Not Validation

Itamar calls out a common shortcut: if the leading competitor ships a feature, teams treat it as validated and rush to build it. He warns this never works, because you shouldn't assume a competitor understands their users any better than you understand yours. Anecdotal signals like one competitor's move are low-confidence evidence, not proof.

  • "The competitor has it, so it's a great idea" is treated as validation in many companies.
  • Copying on that basis never works, in Itamar's experience.
  • You cannot assume a competitor knows what they're doing any more than you do.
  • A single competitor's feature is anecdotal, low-confidence data.

“validation is done let's launch it that's it it's a great idea we need to do it never works honestly”

Itamar Gilad · 41:30

“you should not assume that your competitor actually knows what they're doing anymore than you do”

Itamar Gilad · 41:30
#product#competitors#validation#hot-take
Hot Take44:30

Measure Time-to-Outcomes, Not Time-to-Production

Itamar reframes the classic build-fast-versus-learn tradeoff as a wrong metric. Under uncertainty, speed of shipping bits to production is the wrong thing to optimize — what matters is getting the right bits to production and creating the outcomes you need. He argues the evidence-guided method is actually faster and more resource-efficient than opinion-based building, which wastes effort on the wrong things and learns too late.

  • The right metric is time-to-outcomes, not time-to-production.
  • Under uncertainty, shipping fast to production optimizes the wrong thing.
  • Opinion-based methods waste resources building the wrong things and learning too late.
  • It's a fallacy that learning means building slowly — good teams build and learn at the same time.

“it's not about getting the bits to production it's about getting the right bits to production it's about creating the outcomes that you need”

Itamar Gilad · 45:00
#product#outcomes#speed#hot-take

Explainer· 1

Explainer19:30

Telltale Signs You're Not Actually Evidence-Guided

Many teams assume they run on data but don't. Itamar lists the warning signs: goals that are unclear, too many, vague, or output-focused; missing user-facing metrics (only revenue/business numbers); heavy time sunk into planning and perfect roadmaps; little real experimentation or learning; and disengaged engineers who feel measured only on shipping output.

  • Goals are unclear, plentiful, vague, or framed as output.
  • Metrics are missing or purely business-side (revenue) with no user-facing measures.
  • Lots of time goes into road-mapping, but little experimentation — and where there is, little learning.
  • Engineers are disengaged because they're signaled to just deliver output, not care about users or the business.

“the goals are very unclear either there are many or they're very kind of obscure and vague or they're about output”

Itamar Gilad · 19:30

“the engineers are kind of getting the signal that what they need to do is deliver they're focused on output”

Itamar Gilad · 20:30
#product#metrics#diagnostics#evidence-guided

Story· 3

Story04:30

Google Plus: The Opinion-Based Bet That Missed WhatsApp

Itamar describes joining Gmail in 2011 to wire it into Google Plus, a social-network project Google threw a ~1,000-person division at to counter Facebook. Everyone believed in it, all the early signals looked good, and none of it worked — users didn't want another social network. Worse, while Google poured millions of person-hours in, it missed the far easier social-mobile opportunities like WhatsApp and Snapchat. For Itamar this became the epitome of what he calls opinion-based development.

  • Google created a whole new division around Google Plus, roughly a thousand people at its peak — the size of Android or Docs.
  • Despite early positive indications, people didn't need or use another social network; Gmail later rolled back all the integration and Google Plus shut down in 2018.
  • Google not only bet on the wrong thing, it missed easier wins — WhatsApp and Snapchat were the real threat to Facebook.
  • Itamar admits he was often the PM pushing the big idea; the failure became his turning point toward evidence-guided thinking.

“none of it worked turned out people actually didn't need another social network people didn't love it people didn't use it”

Itamar Gilad · 06:00

“at its peak it was about a thousand people inside”

Itamar Gilad · 07:30
#product#google#opinion-based#strategy
Story10:00

The Tabbed Inbox: The Idea No One Believed In

The project right after Google Plus was the reverse of it — a tiny idea nobody believed in. Instead of pushing his own plan, Itamar's colleagues pointed out that similar inbox-organizing ideas had already flopped, which sent the team into real user research on the passive users drowning in clutter. Rigorous testing produced the promotions/social tabs, a feature that made no sense to power users but that ~85–88% of the population loved. Gmail now has ~1.8 billion active users, most using the feature.

  • It started as a very small idea nobody believed in, the opposite of the top-down Google Plus.
  • Most colleagues manage their own inbox, so splitting promotions and social sounded like the stupidest idea to them.
  • About 85–88% of users — the passive, cluttered majority — absolutely love it.
  • The difference from Google Plus: the team didn't have high confidence in their opinions, so they used evidence instead of going all in.

“it started as a very small idea that no one believed in”

Itamar Gilad · 10:00

“there's about 85 percent of the population 85 to 88 that absolutely love it and today Gmail has about 1.8 billion active users”

Itamar Gilad · 11:30
#product#gmail#user-research#evidence-guided
Story52:00

Faking Gmail's Tabbed Inbox With Zero Code

To validate the tabbed inbox cheaply, the team ran a Wizard of Oz test: an early version shown to users wasn't really Gmail but a facade of HTML. With user permission, team members manually sorted just the subject and sender of the top 50 messages into the right place while an interviewer distracted the person, then revealed a tidy inbox. Users said "wow, this is very cool" — evidence gathered without a single line of code written.

  • The demo was an HTML facade, not real Gmail.
  • Team members hand-sorted the top ~50 messages' subject and sender behind the scenes, with permission.
  • An interviewer distracted users before revealing the sorted inbox.
  • The positive reactions gave enough evidence to justify building it — with no code written.

“it wasn't really Gmail it was just a facade of HTML and behind the scenes and according to the permissions that the users gave us…”

Itamar Gilad · 52:30

“there wasn't a single line of code that's written”

Itamar Gilad · 53:00
#product#testing#wizard-of-oz#gmail

Q&A· 1

Q&A18:30

How to Challenge a Boss's Idea With Evidence

Asked how someone at a less-open company can push back on a founder or exec, Itamar's tactical answer is to run a quiet experiment and bring back real data. One of two things happens: they get furious and tell you to get back to work — in which case that person is unreasonable and you should start job-hunting — or, far more commonly, they are pleasantly surprised and change their mind, exactly as Steve Jobs did when shown Apple could build a phone.

  • Run a secret experiment and come back with hard data rather than arguing opinion-to-opinion.
  • Outcome one: they get mad and dismiss you — a sign the person is unreasonable and you may need to look elsewhere.
  • Outcome two (more common): they are pleasantly surprised and flip their view.
  • In opinion-based debates the most senior person's opinion usually wins; evidence empowers junior and mid-level people to challenge that.

“either they get extremely mad at you and they tell you to get back to work and to do what you were told and in…”

Itamar Gilad · 18:30
#product#leadership#influence#evidence-guided

Tool· 1

Tool1:08:30

ElevenLabs for Cloning Your Own Voice

Itamar's recently-discovered favorite product is ElevenLabs, an AI voice company. Beyond having the best synthetic voices he's heard, it can replicate your own voice into a reusable signature — useful for narrating an audiobook or an online course. He jokes it's part of his "retirement plan" to assemble AI components that can eventually stand in for him.

  • ElevenLabs produces the best synthetic voices he's heard.
  • It can clone your own voice into a personal voice signature.
  • Practical uses: narrating an audiobook or online course.
  • Framed half-jokingly as building an AI version of himself.

“there's a company called 11 Labs uh that do voices and like the best voices synthetic voices you heard but they can also replicate your…”

Itamar Gilad · 1:09:00
#tools#ai#voice#elevenlabs

Takeaway· 3

Takeaway43:00

Don't Test Everything — Know When to Stop

Being evidence-guided doesn't mean testing every change to the hilt. Itamar's counterbalance: match investment to the confidence you need, and recognize when expert opinion alone is enough. Low-risk, low-visibility changes — like reordering settings no one sees — can just be launched. Part of the skill is knowing when to stop climbing the confidence meter.

  • Tie your investment to the level of confidence you actually need.
  • Some changes need no test — expert opinion suffices when risk and visibility are low.
  • Reordering settings nobody notices is a launch-without-testing case.
  • Knowing when to stop matters as much as knowing how to build confidence.

“some ideas you don't need to test sometimes expert opinion is enough if you're just changing the order of the settings no one sees this…”

Itamar Gilad · 43:00
#product#testing#prioritization#takeaway
Takeaway47:30

The Two Types of Companies This Helps Most

Itamar identifies two audiences that respond most strongly to evidence-guided methods. First, companies emerging into modern product development — they have PMs, OKRs, and Agile and are starting to experiment but can't yet stitch it together. Second, companies that used to be evidence-guided and regressed, often after a change of management or culture, and now need to rekindle a lost spirit.

  • Type one: companies moving into modern product management, doing bits of experimentation but struggling to unify it.
  • In those firms every CPO ends up building their own little framework.
  • Type two: once-evidence-guided companies that regressed, which happens too often.
  • Regression usually follows a change of management or culture, requiring a deliberate rediscovery.

“the other type is those companies that used to be evidence guided and they regressed and that happens way too often”

Itamar Gilad · 47:30
#product#organizations#transformation#takeaway
Takeaway1:09:30

Strive Not to Be a Success, But to Be of Value

Itamar's guiding motto, attributed to Albert Einstein, is to strive to be of value rather than to be a success. He ties it to his whole value-exchange philosophy for products and companies. Lenny extends it to content creators: chasing followers and success fails, while delivering genuinely valuable, interesting work is the real signal that resonates with others.

  • The motto: strive not to be a success but to be of value.
  • It loosely underpins Itamar's value-exchange concept for products.
  • Applied to content: creators chasing success and followers usually fail.
  • The signal for value is whether you yourself find it interesting and valuable.

“strive not to be a success but to be of value”

Itamar Gilad · 1:09:30
#mindset#value#motto#takeaway