“it's called mistakes were made but not by me”
Mistakes Were Made (But Not by Me)
By Carol Tavris and Elliot Aronson
1 recommend/use · 1 sourced episode
Every sourced reference
Short attributed excerpts only. Timestamps are approximate.
Related frameworks
False Positive Risk and the Replication Protocol
A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results
Incremental OFAT Over Big-Bang Redesign
Decompose big redesigns into one-factor-at-a-time tested steps because ~80% of ideas fail
Institutional Memory of Experiments
Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas
Sample Ratio Mismatch (SRM) Validity Check
Verify your traffic split matches its design before trusting any experiment result
The 200K-User Experimentation Readiness Rule
Below ~200K users detect only big effects; use the wait to build platform and culture
The Overall Evaluation Criterion (OEC)
Define one success metric with countervailing guardrails that predicts long-term lifetime value
Twyman's Law: Investigate the Too-Good-to-Be-True Result
Any surprising or extreme experiment result is probably wrong until you've hunted for the flaw
People in these episodes
Related resources
- Airbnb
- Amazon
- Apple Podcasts
- Bing
- Blink cameras
- Chernobyl
- Eppo
- GoodUI
- Google Docs
- Hard Facts, Dangerous Half-Truths, and Total Nonsense
- Hotmail
- Lenny's Podcast website
- Maven
- Microsoft
- Mixpanel
- MSN
- Netflix
- Optimizely
- PowerPoint
- Ronny Kohavi’s live cohort-based experimentation course
- Round
- Spotify
- Trustworthy Online Controlled Experiments
- Windows Indexer
Spot an error or want this page removed? Request a correction or removal.