LLenny's Podcast
← All episodes
Ramesh Johari (Stanford professor, startup advisor)09 November 2023

Marketplace lessons from Uber, Airbnb, Bumble, and more

6Frameworks
15Insights

Frameworks in this episode

Innovation4 steps

Fairer Marketplace Rating System Design

Fight rating inflation and averaging bias with renormed labels, priors, blind reviews, and the sound of silence.

Read →
Innovation5 steps

Hypothesis-Driven Experimentation: Learning as a Win

Reframe experiments from winners/losers to hypotheses; run more, run shorter, and carry priors forward.

Read →
Strategy4 steps

Marketplace Whack-a-Mole: Managing Winners and Losers

Most consequential marketplace changes create winners and losers; manage the trade, don't chase a metric.

Read →
Strategy4 steps

Predict vs. Decide: Think in Differentials

Prediction finds correlations; decisions need causal differences. Optimize the lift, not the level.

Read →
Entrepreneurship4 steps

Start Bespoke, Become a Marketplace Later

A marketplace never starts as a marketplace; win with a pre-liquidity value prop first.

Read →
Entrepreneurship4 steps

The Scaled-Liquidity Smell Test

Before calling yourself a marketplace, prove you have scaled liquidity on both sides.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster1:04:30

Averaging Reviews Quietly Punishes Newcomers

Simply averaging ratings feels natural but has serious distributional consequences for who wins and loses. An established seller with 10,000 reviews is unaffected by the next one, while a newcomer whose first rating is negative can be sunk, early eBay work found an ~8% immediate revenue hit and higher exit risk. Ramesh's fix is to blend each seller's ratings with a prior belief that gives newcomers a fairer shot.

  • Averaging feels natural but shapes who wins and who loses in the marketplace.
  • A restaurant with 10,000 Yelp reviews is unmoved by the next one; a newcomer isn't.
  • Early eBay research: a first negative rating caused an ~8% immediate revenue hit and predicted exit.
  • Blending a new seller's ratings with a prior belief pulls unlucky early scores up and improves fairness.

“think of a restaurant on Yelp with 10,000 reviews it's irrelevant what the next review is”

Ramesh Johari · 1:05:00

“if your first rating is negative that could actually in immediately cause like an 8% hit”

Ramesh Johari · 1:05:30
#rating-systems#fairness#reviews#ebay

Hot Take· 3

Hot Take17:00

Stop Calling Yourself a 'Marketplace Founder'

Ramesh pushes back on the identity of 'marketplace founder.' Because almost any online business can eventually enable transactions between two sides, every founder is potentially a marketplace founder, and whether to become a platform is a choice made after growth. His example: nobody thought of OpenAI as a marketplace, yet with plugins it now has creators on one side and users on the other.

  • Almost any modern business can be disrupted by online transactions, so any founder could become a marketplace.
  • Becoming a platform is a choice made after you grow, not an identity you start with.
  • OpenAI didn't set out to be a marketplace but its plugins made it a two-sided one.
  • Whether the company believes it or not, plugin creators plus users equals a marketplace.

“it means literally any founder is a Marketplace founder”

Ramesh Johari · 17:00

“no one in their right mind would have thought of open AI as a marketplace right but open AI is a marketplace now”

Ramesh Johari · 17:30
#marketplaces#founders#platforms#openai
Hot Take40:00

The Language of 'Wins' Is Quietly Ruining Your Experiments

Companies that go all-in on experimentation tend to become risk-averse: data scientists get judged on how many 'wins' they had per quarter, which pushes them toward incremental tests run for too long. Ramesh argues (citing a Microsoft 'AB testing with fat tails' paper) that a big, risky experiment that 'fails' can still teach you a lot, so the goal should be learning, not winning and losing.

  • Going all-in on experiments creates an incentive to be incremental and to run tests too long.
  • Judging data scientists by quarterly 'wins' makes them avoid risky, high-variance tests.
  • A Microsoft paper on 'AB testing with fat tails' argues for trying more, riskier things and running them shorter.
  • A badge experiment that 'fails' still teaches you how attention and inventory get reallocated.

“the culture that says it's okay to fail big actually requires changing the terminology of winds”

Ramesh Johari · 43:00

“you really want to think not in terms of winning and losing but learning so learning is a win”

Ramesh Johari · 45:00
#experimentation#ab-testing#incentives#culture
Hot Take1:09:00

AI Makes Humans More Important to Data Science, Not Less

Against the expectation that LLMs will automate data science away, Ramesh argues the opposite. AI massively expands the frontier of hypotheses, explanations, and things you could test, from 10 marketing creatives to a thousand. That explosion puts MORE pressure on humans to funnel it down and decide what actually matters, making people more central to the productive data-science loop.

  • AI makes coding, visualization, and dashboards faster, but that's the shallow part.
  • Its bigger effect is exploding the number of hypotheses and explanations you can generate.
  • That explosion increases the human's job of funneling down to what matters.
  • Humans have become far more important to the productive data-science loop, not less.

“what AI has done for us is it's massively expanded the Frontier of things we could think about”

Ramesh Johari · 1:09:30

“humans have actually become far more important to the productive data science Loop not far less”

Ramesh Johari · 1:11:00
#ai#data-science#future-of-work#experimentation

Explainer· 5

Explainer05:30

What a Marketplace Actually Sells Isn't Rooms or Rides

Ramesh reframes what a marketplace business is: Airbnb doesn't sell rooms and Uber doesn't sell rides. What they sell is the removal of friction, what economists call transaction costs, the cost of finding a willing counterparty. A consequence is that both sides of the market are the platform's customers, because both depend on it to take that friction away.

  • The platform sells the removal of friction, not the underlying room or ride.
  • Economists call these frictions transaction costs; markets fail in their presence.
  • The real value is answering 'who is out there, willing, right now?'
  • Both buyers and sellers are the platform's customers, since both depend on it.

“both sides of the marketplace are the customers of the platform”

Ramesh Johari · 07:30

“I think this concept that we're making money by taking transaction costs away is such a fundamental idea that's misunderstood around marketplaces”

Ramesh Johari · 08:00
#marketplaces#transaction-costs#business-model#economics
Explainer18:00

Disintermediation: When Your Marketplace Gets Cut Out

Early monetization choices can tie a platform's hands later. Ramesh explains disintermediation, where the two matched parties no longer need the platform, using an Ikea assembly worker who handed over a business card to be booked directly next time. He contrasts Substack (which deepened its social contract by driving demand to writers) with eBay (which broke its social contract by piling on seller fees).

  • Taking a flat cut of every transaction stops making sense once trust is established and the platform adds less value.
  • Disintermediation is when matched parties cut the platform out, like a worker leaving a business card.
  • Substack expanded its business by driving subscriber demand to writers, amplifying the social contract.
  • eBay's escalating seller fees broke a social contract sellers had built their livelihoods on.

“here's my business card ever need me again just call the number on the back”

Ramesh Johari · 19:30

“that's a Breaking of a social contract that's been developed over a very long time”

Ramesh Johari · 22:00
#marketplaces#disintermediation#monetization#pricing
Explainer32:00

Why Sending Promos to Your Highest-LTV Customers Is Wrong

Ramesh's core data-science lesson: prediction is not the same as decision-making. A machine-learning model that predicts high lifetime value tells you who is already valuable, but the real question is how much MORE a customer will spend BECAUSE you sent the promotion, a differential, not an absolute. Prediction is about correlation; good decisions are about causation.

  • ML models pick up on patterns in past data; that's prediction, which is inherently correlational.
  • Sending promos to the highest-LTV customers is safe reputationally but usually the wrong decision.
  • The right question is the incremental spend caused by the promotion, not the customer's absolute LTV.
  • Decisions require causal thinking; a CMO in his class instinctively picked the wrong (correlational) answer.

“predicting what their lifetime value is isn't really the question the question is how much more are they going to spend on my platform because…”

Ramesh Johari · 33:00

“prediction is inherently about correlation but when we ask people to make decisions we're asking him to think about causation”

Ramesh Johari · 34:30
#data-science#causation#machine-learning#decisions
Explainer50:00

Marketplace Management Is a Game of Whack-a-Mole

Because a marketplace has finite attention and inventory, most consequential changes create winners and losers rather than expanding the pie. Ramesh describes fixing one side's experience only to hurt the other, month after month. Quoting former Upwork CFO Sasha Salen, he says the real skill is judging whether the winners you create matter more to your business than the losers.

  • A lot of marketplace management is just moving attention and inventory around.
  • Fixing one side's experience often degrades the other side's, in a repeating cycle.
  • Many consequential changes create winners and losers instead of expanding the pie.
  • The job is deciding whether the winners you've created matter more than the losers.

“a lot of marketplace management is moving attention and inventory around”

Ramesh Johari · 51:00

“many of the changes that are most consequential create winners and losers”

Ramesh Johari · 51:30
#marketplaces#trade-offs#management#inventory
Explainer1:02:30

Why Marketplace Ratings Always Inflate, and How to Renorm Them

Ramesh explains rating inflation, the well-documented tendency (studied by MIT's John Horton) for median ratings to creep up over time due to reciprocity and norming. As everyone drifts toward five stars, four stars starts to feel like a punishment. His fix is to renorm the labels, e.g. making the top rating 'exceeded expectations' or asking people to compare against a past great experience, as Airbnb did.

  • Median ratings inflate over time on marketplaces like oDesk, Uber, and others.
  • Reciprocity (people don't want to be mean) and norming both push ratings up.
  • Once ratings normalize high, giving four stars feels like screwing someone over.
  • Renorming labels ('exceeded expectations', compare to a past great stay) makes honest ratings easier.

“over time you see the median rating inflating”

Ramesh Johari · 1:03:00

“the top rating is actually exceeded expectations”

Ramesh Johari · 1:04:00
#rating-systems#reviews#marketplaces#design

Story· 3

Story46:00

The Airbnb Superhost Badge Tested Flat, and Was Still a Great Idea

Lenny recounts launching Superhost at Airbnb over a data scientist's fierce objection that badging random listings would wreck the carefully built ranking algorithm. The A/B test showed essentially no impact at all, only a slight lift in host satisfaction. Yet in hindsight Lenny can't imagine Airbnb without it, illustrating that not every good marketplace change trips a metric you can measure short-term.

  • The data team feared Superhost would destroy the ranking algorithm by badging listings it hadn't optimized for.
  • The experiment showed no measurable impact on the business, only that hosts felt more satisfied.
  • In hindsight it feels like it made the marketplace better despite no initial evidence.
  • Some valuable changes don't move a short-term metric, especially when they rebalance inventory.

“we ran an experiment showing the badge to some people and some not actually it was no no impact at all”

Lenny · 46:30

“it was a great idea I'm really happy I can't even imagine airb be without that even though there's no evidence at least initially that…”

Lenny · 49:30
#airbnb#experimentation#badges#superhost
Story57:30

The Marketing Manager Who Ran an Unauthorized Holdout

Ramesh's favorite anecdote: a real-estate platform's marketing manager secretly held out a group of visitors from all his ad innovations. At year-end the holdout had cost a couple of million dollars, and it wasn't authorized. His defense: now you know what my team is worth, and you'd never have that answer otherwise. The point (with a Seinfeld reference) is that learning has a cost you must be willing to pay.

  • The manager kept an unauthorized holdout group that saw none of the year's ad innovations.
  • The holdout cost the company roughly a couple of million dollars.
  • His justification: it proved the team's value, an answer no other method could produce.
  • Once you know the answer it feels obvious, but at decision time you must pay to learn it.

“now you know what my team's worth and number two you would never have had that answer unless I'd done that on my own”

Ramesh Johari · 58:30

“that idea that you have to pay to learn is is again it's a cultural thing”

Ramesh Johari · 1:01:00
#experimentation#holdout#learning#roi
Story1:07:00

Double-Blind Reviews and the 'Sound of Silence'

Lenny describes launching double-blind reviews at Airbnb, where neither party sees the other's review until they leave their own; the biggest effect was a jump in review rate, which yielded more data. Ramesh connects this to the 'Sound of Silence' concept: a lot of information lives in the ratings that are never left, and Berkeley's Steve Tadelis found that counting non-reviews ('effective percent positive') better predicts seller performance.

  • Double-blind reviews hide each party's review until the other is submitted, intended to boost honesty.
  • The biggest measured impact was a higher review rate, which produced more data.
  • The 'Sound of Silence': there's a lot of information in the ratings people choose NOT to leave.
  • Steve Tadelis's 'effective percent positive' (counting non-reviews) better predicts seller performance.

“it turned out the biggest impact was review rate went up”

Lenny · 1:07:30

“The Sound of Silence which is this idea that that there's a lot of information in in ratings that are not left”

Ramesh Johari · 1:07:30
#rating-systems#airbnb#reviews#research

Tool· 1

Tool1:11:30

Ramesh Johari's Three Book Recommendations

In the lightning round, Ramesh recommends three books: 'How to Lie with Statistics' by Darrell Huff (1954), a tiny, fun read for anyone who likes data; the writing of Berkeley statistician David Freedman, known for 'shoe-leather statistics' and emphasis on understanding the process that generates your data; and 'Four Thousand Weeks' by Oliver Burkeman, a mildly stoic take on prioritizing a finite life.

  • 'How to Lie with Statistics' by Darrell Huff (1954), a short, fun read for any data lover.
  • David Freedman's writing on 'shoe-leather statistics', get on the ground and understand your data's process.
  • 'Four Thousand Weeks' by Oliver Burkeman, a stoic-flavored book on prioritizing a finite life.

“how to lie with Statistics it's a tiny book by Daryl Huff from 1954”

Ramesh Johari · 1:11:30
#books#recommendations#statistics#productivity

Takeaway· 2

Takeaway12:30

A Marketplace Business Never Starts as a Marketplace

The biggest failure mode Ramesh sees is founders thinking too much like a marketplace before they actually are one. Using UrbanSitter (which began by simply accepting credit-card payments for babysitters) and oDesk (which began by verifying remote workers' hours), he argues the early value proposition must solve a concrete problem in a world without scaled liquidity, not the friction of two sides finding each other.

  • The biggest failure mode is behaving like a marketplace before you have liquidity.
  • UrbanSitter's first wedge was accepting credit-card payments for babysitters, then Facebook-based trusted intros.
  • oDesk's first wedge was tools letting remote workers prove they worked the hours they claimed.
  • You can't solve 'help you find matches' when you only have three sellers on the platform.

“they think too much about a Marketplace before they're a Marketplace that in my view is the biggest failure mode”

Ramesh Johari · 12:30

“a Marketplace business never starts as a Marketplace business because what we think of as a Marketplace business is something which at scale is removing…”

Ramesh Johari · 15:00
#marketplaces#liquidity#startups#cold-start
Takeaway22:30

The Scaled-Liquidity Smell Test for 'Am I a Marketplace?'

Ramesh offers a limit test for anyone claiming to be building a marketplace: do you have scaled liquidity, lots of buyers AND lots of sellers? If you only have one side, you've won that side and should just lean into scaling it. If you have neither, forget being a marketplace and focus on unit economics, like whether you could just hire people (a la an early DoorDash).

  • The smell test: do you have a lot of buyers AND a lot of sellers, or just one, or neither?
  • If you only have one side, there's no shame in just scaling that side.
  • If you have neither, don't worry about being a marketplace at all.
  • Uber seeded new cities by handing out free-ride coupons, using its subsidized driver side to attract riders.

“do I have a lot of buyers and a lot of sellers on my platform or do I only have one of these two or…”

Ramesh Johari · 23:30

“you can call yourself whatever you want to call yourself but at this moment in time you're not a Marketplace”

Ramesh Johari · 23:30
#marketplaces#liquidity#growth#uber