LLenny's Podcast
← All episodes
Judd Antin (Airbnb, Meta)04 January 2024

The UX research reckoning is here

8Frameworks
14Insights

Episode overview

Judd Antin argues that UX research is undergoing a reckoning because many organizations built reactive service functions that produce interesting but insufficiently actionable work. He proposes shifting effort toward strategic macro research and fast, business-linked micro research while embedding researchers throughout product development. The conversation also covers researcher hiring, business literacy, user-centered performance, experimentation, NPS, and productive partnerships among research, product, design, engineering, and data science.

Key ideas

  • UX research's current difficulties reflect a broken organizational system, not the disappearance of the need for research.
  • Judd divides research into macro, middle-range, and micro work, arguing that undirected middle-range research is often interesting but weakly tied to business impact.
  • Researchers should be embedded from the beginning of product work instead of operating as a reactive service requested near launch.
  • Strong researchers combine formative research, evaluative research, survey design, applied statistics, and technical skills such as SQL, dashboarding, or prompt engineering.
  • “User-centered performance” occurs when teams symbolically seek validation or customer proximity without being willing to change their decisions.
  • Research should try to falsify assumptions, complement intuition, and explain the how and why behind causal experimental results.
  • Researchers need business fluency, strong communication, ruthless prioritization, and shared success metrics with their product partners.
  • Judd argues that customer-satisfaction measures generally have better survey properties than NPS.

Transcript available · source text is retained privately and is not published

Frameworks in this episode

Marketing4 steps

Kill NPS, Ask CSAT

NPS is the marketing industry marketing itself — a satisfaction question has better data properties.

Read →
Strategy4 steps

Macro / Middle-Range / Micro Research Portfolio

Sort every research question into three altitudes and starve the middle one that eats your team's time.

Read →
Influence4 steps

The Business-Fluency Scalpel for Researchers

Learn the company's financial language so your insights land in the funnel, not in a deck.

Read →
Strategy3 steps

The Dogfooding Boundary Rule

Use your own product walkthrough to generate the issue list — never to prioritize it.

Read →
Leadership5 steps

The Embedded Insights Partner Model

Success metric: they won't have the meeting without you.

Read →
Leadership4 steps

The Five Tools Researcher

A modern researcher carries five methods, not one — and you hire for the Swiss army knife.

Read →
Self-Mastery4 steps

The Gut-Check Protocol

Intuition is pattern matching and bias in the same organ — check it with System 2 and diverse guts.

Read →
Mindset4 steps

User-Centered Performance Detector

Spot the rituals where a team signals customer obsession without any intent to change the decision.

Read →

People & resources mentioned

Attributed to the moment in the episode. Timestamps are approximate.

People · 11

  • Lenny RachitskyMentionstoday we've got another very special compilation episode something I've been pulling on more and more with the podcast and the newsletter

    “today my guest is Jud Anon”

    ▶ Watch at ~0:45approx.
  • Judd AntinMentions“Jud helped build the user research practice at Facebook he was a longtime head of research at Airbnb”

    “Jud helped build the user research practice at Facebook he was a longtime head of research at Airbnb”

    ▶ Watch at ~0:52approx.
  • Matt GallivanMentions“Matt Gallivan who now leads research at slack”

    “Matt Gallivan who now leads research at slack”

    ▶ Watch at ~4:33approx.
  • Janna BrayMentions“Janna Bray who leads research at notion”

    “Janna Bray who leads research at notion”

    ▶ Watch at ~4:38approx.
  • Rebecca GrayMentions“Rebecca gray who leads research at Fair”

    “Rebecca gray who leads research at Fair”

    ▶ Watch at ~4:42approx.
  • Henry FordMentionsthen Henry F came along or Steve Jobs came along and they had this invention

    “it turns out a totally apocryphal Henry Ford quote”

    ▶ Watch at ~42:18approx.
  • Duncan WattsMentions“the author is a sociologist at upen named Duncan watts”

    “the author is a sociologist at upen named Duncan watts”

    ▶ Watch at ~43:25approx.
  • Patrick CollisonMentions“there's this tweet by Patrick hollison that I've brought up a couple times on this podcast”

    “there's this tweet by Patrick hollison that I've brought up a couple times on this podcast”

    ▶ Watch at ~44:55approx.
  • Brian CheskyMentionsBrian chesky at Airbnb there was a whole article where we unpacked this moment in their product development cycle

    “if I'm talking to you Lenny it's different than I'm talking to Brian chesky”

    ▶ Watch at ~1:03:04approx.
  • Barbara KellermanMentions“a business book by Barbara Kellerman called bad leadership”

    “a business book by Barbara Kellerman called bad leadership”

    ▶ Watch at ~1:09:03approx.
  • EpictetusMentions“epicus was writing about this you know BC”

    “epicus was writing about this you know BC”

    ▶ Watch at ~1:12:15approx.

Resources · 26

  • FacebookMentionscompany · Meta

    “Jud helped build the user research practice at Facebook”

    ▶ Watch at ~0:52approx.
  • AirbnbMentionscompany · Brian Chesky, Joe Gebbia, and Nathan Blecharczyk

    “he was a longtime head of research at Airbnb”

    ▶ Watch at ~0:57approx.
  • TealRecommendssoftware · Teal

    “teal is the all-in-one platform you need to run a more streamlined and efficient job search”

    ▶ Watch at ~2:48approx.
  • VantaMentionssoftware · Christina Cacioppo

    “this episode is brought to you by vanta helping you streamline your security compliance to accelerate your growth”

    ▶ Watch at ~3:12approx.
  • SlackMentionssoftware · Slack Technologies

    “Matt Gallivan who now leads research at slack”

    ▶ Watch at ~4:33approx.
  • NotionMentionssoftware · Notion Labs

    “Janna Bray who leads research at notion”

    ▶ Watch at ~4:38approx.
  • RobinhoodMentionscompany

    “Celeste Ridin who leads research at Robin Hood”

    ▶ Watch at ~4:40approx.
  • FaireMentionscompany

    “Rebecca gray who leads research at Fair”

    ▶ Watch at ~4:42approx.
  • DuolingoMentionssoftware · Luis von Ahn and Severin Hacker

    “Hannah Pelley who I think was leading research at dualingo”

    ▶ Watch at ~4:44approx.
  • FigmaMentionssoftware · Figma

    “Luis barel who leads research at figma”

    ▶ Watch at ~4:48approx.
  • WealthfrontMentionscompany

    “gnome who was leading research at wealth front”

    ▶ Watch at ~4:51approx.
  • The User Research Reckoning Is HereCoinedother · Judd Antin

    “you wrote this piece that was titled the user research Reckoning is here”

    ▶ Watch at ~5:15approx.
  • IDEOMentionscompany

    “design thinking which which idio popularized”

    ▶ Watch at ~28:37approx.
  • UC BerkeleyUsescompany

    “I taught a class at UC Berkeley this semester on leadership”

    ▶ Watch at ~29:30approx.
  • WikipediaMentionswebsite · Wikimedia Foundation

    “there's a list of 151 of them on Wikipedia”

    ▶ Watch at ~30:03approx.
  • Ahrefs Webmaster ToolsRecommendstool · Ahrefs

    “hre's webmaster tools will show you keywords that you rank for and backlinks that you can get”

    ▶ Watch at ~36:58approx.
  • Everything Is Obvious: Once You Know the AnswerRecommendsbook · Duncan Watts

    “the title is everything is obvious if you already know the answer and it's about hindsight bias”

    ▶ Watch at ~43:29approx.
  • FacebookUsescompany · Meta

    “one thing that you can do on Facebook is hide a post”

    ▶ Watch at ~53:06approx.
  • Windows 11Mentionssoftware · Microsoft

    “How likely are you to recommend Windows 11 to your friends and family”

    ▶ Watch at ~1:05:00approx.
  • Bad LeadershipRecommendsbook · Barbara Kellerman

    “a business book by Barbara Kellerman called bad leadership”

    ▶ Watch at ~1:09:03approx.
  • The Last of UsRecommendstv show · Craig Mazin and Neil Druckmann

    “we recently watched the last of us and it blew our mind”

    ▶ Watch at ~1:09:57approx.
  • The Last of UsRecommendstv show · Craig Mazin and Neil Druckmann

    “if you are a person who plays video games and you haven't played The Last of Us play it”

    ▶ Watch at ~1:10:04approx.
  • Belay GlassesUsesproduct

    “they make these glasses which are called Bay glasses and they have an angled mirror embedded in the lens”

    ▶ Watch at ~1:11:15approx.
  • The 7 Habits of Highly Effective PeopleMentionsbook · Stephen R. Covey

    “I learned that lesson from Seven Habits of Highly Effective People”

    ▶ Watch at ~1:12:35approx.
  • Apple PodcastsMentionssoftware · Apple Inc.

    “you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”

    ▶ Watch at ~1:14:10approx.
  • SpotifyMentionssoftware · Spotify

    “you can subscribe to the show on Apple podcast Spotify or your favorite podcast app”

    ▶ Watch at ~1:14:10approx.

Spot an error or want something removed? Request a correction or removal.

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 2

Myth Buster33:30

The Tropes About Research That Drive Researchers Nuts

Antin lists the untrue things PMs say about research. 'Research is too slow' is false — good research can take a day, a week, or a month, and it's slower to get it wrong and fix it. 'I can do my own research' ignores that anyone can talk to a user but one idiosyncratic user isn't insight. 'AB test everything' fails because tests rarely tell you why a result happened.

  • 'Research is too slow' is BS — good research speeds you up, and getting it wrong is slower
  • Anyone can talk to a user, but that alone is not research or insights work
  • One user can be powerful but also idiosyncratic; researchers situate it correctly
  • AB tests are near-causal but rarely explain how or why a result happened

“good research doesn't slow us down it speeds us up”

Judd Antin · 34:00

“anyone can talk to a user that does not constitute research or insights work”

Judd Antin · 39:00
#research#myths#ab-testing#product
Myth Buster42:00

The Henry Ford 'Faster Horses' Quote Is Apocryphal — And Misunderstands Research

Antin debunks the ubiquitous Henry Ford 'if I'd asked my users they'd have wanted a faster horse' quote — to the best of our knowledge, Ford never said it. Beyond being apocryphal, it makes researchers angry because it misrepresents what they do. A good researcher never simply asks customers what they want; that's the mark of a bad researcher.

  • The famous Henry Ford quote is, to the best of our knowledge, apocryphal
  • It makes researchers mad because it isn't what research actually is
  • A researcher who asks customers what they want is a bad researcher
  • Good research uncovers underlying needs, not literal feature requests

“everyone loves to quote that that it turns out a totally apocryphal Henry Ford quote”

Judd Antin · 42:00

“a researcher who's going to ask customers what they want is a bad researcher”

Judd Antin · 43:00
#research#myths#henry-ford#insights

Hot Take· 5

Hot Take05:30

Why the UX Research Discipline of the Last 15 Years Is Dying

Antin explains that the wave of UX research layoffs is not just a downturn but a signal that the system around research is broken. Research, he argues, has not been driving the value or impact it could, largely because of how it's positioned and integrated inside companies. He frames the moment as a 'Reckoning' — a chance to take stock and evolve rather than an obituary for the field.

  • UX and UX research were hit unusually hard by layoffs, which Antin reads as a systemic signal
  • The core problem is that research isn't driving the value or impact it should or could
  • Much of the blame is structural — how research is integrated and positioned, not researcher talent
  • 'Reckoning' was chosen to mark a moment to take stock, though he admits it added unwanted drama

“it's a sign that maybe the system is a little more broken than we think and that research is not driving the value or impact…”

Judd Antin · 08:00
#ux-research#layoffs#career#product
Hot Take21:00

Researchers Need to Be Way More Profit-Focused

Antin's most controversial point is that researchers must find the overlap between what users need and what the business needs, and be explicit about it. His practical advice: read the quarterly report, listen to the shareholder call, and learn the OKRs, metrics, and conversion funnel cold. That fluency lets a researcher point to the exact spot in the funnel where research will drive impact.

  • Focus on the overlapping venn of user value and business/profit
  • Read the last quarterly report and shareholder call to learn the language of the business
  • Know the OKRs, metrics, and conversion funnel back and forth
  • Business fluency lets researchers target research to a specific high-impact opportunity

“there's an overlapping vent where you have the user and profit or the business and what researchers need to do is be way more explicit…”

Judd Antin · 21:30

“did you read the last quarterly report if it's a public company did you listen to the shareholder call”

Judd Antin · 22:00
#business#research#profit#strategy
Hot Take23:30

User-Centered Performance: Faking Customer Obsession for Show

Antin coins 'user-centered performance' — customer obsession that is symbolic rather than focused on learning. The classic example is a PM asking for a quick user study at the end of a process just to validate assumptions, which is checking a box, not seeking to be wrong. His counter-mantra is that good teams don't validate, they falsify — they actively look to be proven wrong.

  • User-centered performance is symbolic customer obsession done to signal, not to learn
  • The 'quick user study to validate our assumptions' request is too late to matter
  • Executive listening sessions and focus-group requests are often ~97% performance
  • The right mindset is to falsify, not validate — to actively want to be wrong

“it refers to um customer Obsession or user centered practice that is symbolic rather than focused on learning”

Judd Antin · 24:00

“we don't validate we falsify right we are looking to be wrong”

Judd Antin · 26:00
#user-centered#bias#product#mindset
Hot Take1:03:30

NPS Is Garbage — Ask Customer Satisfaction Instead

Antin delivers a strong hot take that NPS is the best example of the marketing industry marketing itself. He argues the likelihood-to-recommend question is methodologically bad — an 11-point often-unlabeled scale, poor on mobile below the fold, and a fundamentally flawed loyalty proxy. A simple customer satisfaction (CSAT) question has better data properties and correlates more strongly with business outcomes.

  • Survey scientists consider NPS a garbage-in, garbage-out metric
  • The 0-10 scale, unlabeled points, and 11 items all hurt precision, especially on mobile
  • Likelihood-to-recommend is a flawed proxy for loyalty
  • CSAT is more precise and more correlated to business outcomes; NPS is idiosyncratic and not comparable across companies

“NPS is the best example of the marketing industry marketing itself”

Judd Antin · 1:03:30

“don't ask NPS ask customer satisfaction”

Judd Antin · 1:05:30
#nps#surveys#metrics#csat
Hot Take1:07:00

Dog-Food Your Product, But Remember You Are Nothing Like the User

Antin offers a contrarian take on PMs relying on their own product walkthroughs. Everyone should dog-food their product to build a list of potential issues, but the danger is trusting your intuition about what matters — because you are nothing like the user. Prioritizing which issues actually matter, and for whom, requires context of use you can't get from your own usage.

  • Everyone should dog-food their own product to surface potential issues
  • The trap is relying on your intuition about what's good or bad in the product
  • You are nothing like the user in ways that bias you and that you can't recognize
  • Be extremely wary of using your own opinion to prioritize which issues matter

“you are nothing like the user you are nothing Li them in ways that will bias the way you think about what's good and bad…”

Judd Antin · 1:07:30
#dogfooding#bias#product-management#intuition

Explainer· 1

Explainer30:30

How to Actually Check Your Gut: Get Different Guts in the Room

Antin explains that intuition is where cognitive biases live, so leaders should check their gut using System 2 analytic thinking and the wisdom of the crowd. Crucially, the wisdom of the crowd only works when people bring diverse sources of information and judgment — not the same inputs. A researcher is often the person best suited to lead those diverse, direct, and kind decision discussions.

  • Intuition/gut is where cognitive biases and blind spots live
  • Check the gut with slow, methodical System 2 analytic thinking
  • Wisdom of the crowd only works when participants bring diverse information and judgment
  • Researchers are well-positioned to lead these evidence-plus-intuition discussions

“the wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table”

Judd Antin · 30:30

“if you want to check your gut get a bunch of different guts together”

Judd Antin · 31:00
#decision-making#bias#intuition#teams

Story· 2

Story35:30

The Multi-Million Dollar Button: How Seven Characters Made Millions

Antin tells the 'multi-million dollar button' story from Airbnb. Research revealed users weren't going down the purchase funnel because the call-to-action text made them afraid the button would immediately initiate a purchase. Changing about seven characters of button text lifted conversion by roughly 1% and made Airbnb millions — a fast, high-value example of micro research.

  • Users avoided the funnel because the CTA felt like it would trigger a purchase
  • The fix was a seven-character change to the button text, with UX writing help
  • The change added about 1% conversion, which is very hard to do
  • Discovered essentially overnight — a ~48-hour micro research win

“we basically changed seven characters and made Airbnb millions of dollars”

Judd Antin · 35:30
#airbnb#conversion#case-study#micro-research
Story52:30

The Facebook 'Super-Hiders' and the Power of an N of One

Antin recounts a Facebook study on post quality where engineers wanted to use 'hides' as a signal of bad posts. He found hiding was power-law distributed, so his team interviewed a 'super-hider' behind glass with product leaders watching. She hid every story — not because they were bad, but because she was treating the feed like an inbox-zero to clear, which the infinite scroll made impossible.

  • Engineers proposed using the most-hidden posts as a signal of low quality
  • Hiding turned out to be power-law distributed — a few 'super-hiders' do most of it
  • A live interview showed one super-hider hiding stories she'd already seen, chasing inbox zero
  • An N of one let watching leaders burst their own bubble about what hides meant

“the model she was going for was inbox zero which was like sad because it was infinitely scrolling she would never get there”

Judd Antin · 55:00

“the N of one allowed them to burst their own bubble”

Judd Antin · 55:30
#facebook#case-study#qualitative#signals

Q&A· 2

Q&A57:00

How Many Researchers Does a Company Actually Need?

Asked for the right researcher-to-team ratio, Antin says there's no clean number — it depends on product stage and company maturity. His organizing principle is relationships: staff enough so that people who need a constant research partner have one, and pair researchers deeply rather than spreading them thin. He notes a great researcher makes you go faster, not slower, and drives headcount growth via partners who feel the pain of loss.

  • There's no universal ratio; it depends on product and company stage
  • Organize around relationships — pair a researcher deeply rather than spreading thin
  • Create pain via 'loss' to grow headcount, letting partners argue the case for you
  • A good researcher makes you go faster, answering otherwise-impossible questions
  • On a list of ~20 B2B companies' first 10 employees, only one was a researcher

“a good researcher makes you go faster not slower”

Judd Antin · 59:00

“there is one researcher on that list anywhere anywhere and that's messed up to me”

Judd Antin · 59:00
#hiring#ratio#org-design#startups
Q&A1:10:00

Antin's Favorite Interview Question: Explain It Like I'm Five

In the lightning round, Antin shares his favorite candidate question: think of the most complex topic you recently had to explain, then explain it to him like he's five. He's asked it of VP and C-suite candidates across disciplines, sometimes about quantum computing, music theory, or a complex business decision. He's looking for whether someone can break a complex problem down simply and give intuition fast — a key differentiator between good and great.

  • The question: take the most complex thing you recently explained and explain it like I'm five
  • Used across disciplines and seniority, including VP and C-suite candidates
  • Topics can range from quantum computing to music theory to business decisions
  • Tests the ability to simplify complexity fast — a good-vs-great differentiator

“think of a topic that you had to explain lately that was the most complex and then explain it to me like I'm five”

Judd Antin · 1:10:00
#hiring#interviewing#communication#lightning-round

Takeaway· 2

Takeaway14:30

Research as a Service Function Is the Number One Root Problem

Antin argues the biggest failure is treating researchers as a reactive service function called in at the end of a product process. The fix is to integrate researchers into the process from beginning to end, in the room and building direct relationships, so they can frame the right high-impact questions. He's confident that a great researcher consistently embedded in a product process will drive product improvement, metrics, and growth.

  • Research called in only at the end is reactive and misses the chance to frame the right question
  • As long as research is a service discipline the field stays stuck
  • The short circuit is constant engagement — a researcher in the room from the start
  • A well-integrated great researcher drives product improvement, metrics impact, and growth

“as long as research is a service discipline I think we're going to be stuck in this spot”

Judd Antin · 15:30

“if you take a great researcher and you insert them consistently in a product process I feel confident that researcher will drive uh product Improvement”

Judd Antin · 17:00
#research#product-process#org-design#integration
Takeaway17:30

The Five Tools Every Great Modern Researcher Needs

Antin says the best researchers are multi-method, not just qualitative. He lists five tools: formative/generative research, evaluative (usability) research, rigorous survey design, applied statistics, and a technical skill that used to be SQL but may now be prompt engineering. He argues this evolution away from a purely qualitative discipline is essential.

  • Tool 1: formative/generative, open-ended, ethnographic, innovation-focused research
  • Tool 2: evaluative research such as usability testing
  • Tool 3: rigorous survey design to scale responses
  • Tool 4: applied statistics — you can't operate in an AB-testing world without it
  • Tool 5: technical skills — historically SQL, now possibly prompt engineering

“the best researchers have five tools I think they have five tools and those five tools are”

Judd Antin · 17:30
#research-skills#hiring#surveys#statistics