“it's very hard for Bing to catch up because Bing doesn't have as much data to trade on”
Bing
By Microsoft
1 recommend/use · 2 sourced episodes
Every sourced reference
Short attributed excerpts only. Timestamps are approximate.
“somebody proposed to change the way that ads were displayed on Bing the search engine”
“the biggest Revenue impact of being in all its history”
Related frameworks
False Positive Risk and the Replication Protocol
A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results
Focusing Power on an Addressable Target
Strategy works only when you concentrate a genuine source of power on a target that can actually be moved.
Incremental OFAT Over Big-Bang Redesign
Decompose big redesigns into one-factor-at-a-time tested steps because ~80% of ideas fail
Institutional Memory of Experiments
Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas
Sample Ratio Mismatch (SRM) Validity Check
Verify your traffic split matches its design before trusting any experiment result
Spotting Bad Strategy
Four tells that a 'strategy' is actually empty: no diagnosis, goals-as-strategy, fluff, and incoherent actions.
The 200K-User Experimentation Readiness Rule
Below ~200K users detect only big effects; use the wait to build platform and culture
The Action Agenda (Ambition-Filter Method)
Stop writing a 'strategy.' Filter your ambitions down to one important, addressable challenge and act.
The Crux: Insight Through Immersion
Find the single hardest solvable part of the problem, immerse in it, and let insight emerge.
The Foundry: Facilitating a Group to One Action Agenda
Put the leader plus 7-8 people in a room, surface 25 problems, then focus on the one or two addressable ones.
The Kernel of Good Strategy
Every real strategy has three parts: a diagnosis, a guiding policy, and coherent action.
The Overall Evaluation Criterion (OEC)
Define one success metric with countervailing guardrails that predicts long-term lifetime value
The Value Deny: Finding Opportunity in What You Can't Buy
List what you should be able to buy but can't, or how a familiar product ought to work but doesn't.
Twyman's Law: Investigate the Too-Good-to-Be-True Result
Any surprising or extreme experiment result is probably wrong until you've hunted for the flaw
People in these episodes
Related resources
- Airbnb
- Amazon
- Apple
- Apple Podcasts
- Blink cameras
- Chernobyl
- CommandBar
- Eppo
- Good Strategy/Bad Strategy
- GoodUI
- Google Docs
- Hard Facts, Dangerous Half-Truths, and Total Nonsense
- Harvard Business School
- Hotmail
- IBM
- Intel
- Lenny's Podcast
- Lenny's Podcast website
- Maven
- Microsoft
- Miro
- Mistakes Were Made (But Not by Me)
- MIT
- Mixpanel
- MSN
- Netflix
- Only the Paranoid Survive
- Optimizely
- Playing to Win
- PowerPoint
- Ronny Kohavi’s live cohort-based experimentation course
- Round
- Seven Powers
- SpaceX
- Spotify
- The Crux
- The Crux Book website
- The Wall Street Journal
- Trustworthy Online Controlled Experiments
- Vanta
- Windows Indexer
- Yellowstone
Spot an error or want this page removed? Request a correction or removal.