Fairer Marketplace Rating System Design
Fight rating inflation and averaging bias with renormed labels, priors, blind reviews, and the sound of silence.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 90%
Rating systems are understudied yet decisive for who wins and loses in a marketplace. Two structural problems dominate: rating inflation from reciprocity and norming, and the distributional unfairness of naive averaging, which can end a newcomer's career on one bad review. Johari offers concrete design levers: renorm the labels, blend new ratings with a prior, use double-blind reviews, and read the information in reviews never left.
Origin
Ramesh Johari; references John Horton (MIT/upwork) on rating inflation, Steve Tadelis (Berkeley/eBay) on 'the sound of silence' / effective percent positive, and Airbnb's double-blind review launch.
Core principles
- 01Over time median ratings inflate due to reciprocity (it's costless to ask for a nice rating) and norming (a 4-star starts to feel harsh)
- 02Naive averaging is distributionally unfair: an entrenched seller is immune to new reviews while a newcomer's first negative can be fatal
- 03A first negative rating on eBay caused an immediate ~8% hit and predicted eventual exit
- 04Ratings never left carry information ('the sound of silence'); it's far easier to skip a review than leave a bad one
How to run it
- 1
Renorm the rating labels
Replace generic poor-to-excellent scales with expectation-anchored language so honest signal survives inflation.
Pro tip Make the top rating 'exceeded expectations,' or ask the rater to compare against a past standout experience; people find it easier to say 'good but didn't exceed' than to dock stars.
- 2
Blend new ratings with a prior instead of naive averaging
For newcomers, average their sparse ratings together with a prior belief so one unlucky negative doesn't permanently sink them.
Pro tip A prior that pulls an early rating up a little keeps a newcomer viable alongside established sellers and preserves distributional fairness.
Watch out Some platforms hide ratings until a few accumulate, but averaging bias is the deeper structural issue to fix.
- 3
Use double-blind reviews
Hide each party's review until both have submitted, so ratings aren't anchored to or retaliating against the other side's.
Pro tip Airbnb's double-blind launch, paired with a reminder email, raised review rate — yielding more data as well as more honesty.
- 4
Read the sound of silence
Treat non-reviews as signal; normalize by including ratings that weren't left, not just those submitted.
Pro tip Tadelis's 'effective percent positive' (normalizing by including missing ratings) was far more predictive of a seller's downstream performance.
In the wild
Johari led Airbnb review flows and launched double-blind reviews (you don't see the other person's review until you leave yours), paired with a reminder email prompting people to review to unlock the other's.
→ Review rate rose, producing both more honest reviews and more data to power the marketplace.
Early eBay work showed a seller's first rating being negative caused roughly an 8% immediate hit to expected revenue and, in later research, predicted exit from the platform because work became hard to find.
→ Motivates prior-blended scoring so newcomers survive an unlucky first review.
Common mistakes
Naive averaging of ratings
It makes entrenched sellers immune to new reviews while letting a newcomer's single negative cause an ~8% revenue hit and eventual exit.
Ignoring reviews that were never left
The absence of a review is informative; normalizing only over submitted ratings discards a strong predictor of downstream quality.
Is it for you?
Best for
Marketplace founders and product teams designing or overhauling ratings/reviews and worried about fairness to new supply
Not ideal for
Products with no repeated two-sided interactions or reputation dynamics
From the transcript
“the top rating is actually exceeded expectations”
“if your first rating is negative that could actually in immediately cause like an 8%”
“instead of averaging them I average them together with a prior belief”
“double blind ex uh reviews where don't see the other person's review until you leave your review”
“there's a lot of information in in in ratings that are not left”
From the episode
Marketplace lessons from Uber, Airbnb, Bumble, and more
Ramesh Johari (Stanford professor, startup advisor)