Institutional Memory of Experiments
Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 90%
Running many experiments creates value only if the organization remembers what it learned. Kohavi built institutional memory through searchable experiment histories and a quarterly meeting of the most surprising experiments, preventing the common failure of launching experiments and never summarizing the learnings.
Origin
Kohavi's practice at Microsoft and Airbnb; references good UI pattern libraries (goodui.org by Jakub Linowski) and the Microsoft 'rules of thumb' paper as external memory aids.
Core principles
- 01Many companies launch experiments but never go back and summarize the learnings — a core mistake
- 02Winners get semi-forgotten over time, so successful patterns must be documented and re-surfaced
- 03A surprising experiment is one where the pre-estimate and actual result differ greatly — surface these deliberately
- 04Both surprising winners and surprising losers carry the most learning
- 05Searchable history plus a recurring review feeds the flywheel of experimentation
How to run it
- 1
Keep a searchable experiment history
Store every experiment so anyone can search by keyword and ask 'has anybody tried this?' before spending effort.
Pro tip With tens of thousands of experiments a year, keyword search prevents re-running known ideas.
- 2
Run a quarterly surprising-experiments review
Hold a recurring meeting of the most interesting (not just most successful) experiments, defining surprising as a large gap between expected and actual results.
Pro tip Focus on 'most interesting' rather than 'most successful' to capture instructive losers too.
- 3
Document winners and losers durably
Maintain a deck or repository of successes and failures that survives employee turnover, so learnings persist three years later when people leave.
Pro tip Re-introduce forgotten winners to new teams — institutional amnesia loses proven gains.
Watch out Institutional memory decays; winners that aren't re-surfaced get silently dropped from new designs.
- 4
Leverage external pattern libraries
Supplement internal memory with external resources like the 'rules of thumb' paper and goodui.org, which catalog what tends to work and how often.
Pro tip Pattern libraries showing 'worked 3 of 5 times, big win' can seed a roadmap of high-probability ideas.
In the wild
Kohavi's team proved opening links in a new tab was highly beneficial around 2008 at Microsoft; at Airbnb the practice existed for listings but had been 'semi-forgotten' and not applied to newer designs, so he re-introduced it.
→ Re-applying the forgotten winning pattern produced big improvements again, illustrating the cost of lost institutional memory.
When a relevance-improving indexer change killed laptop battery life, the surprising loser was documented so future iterations would account for the CPU/battery factor.
→ The organization gained a durable design constraint instead of rediscovering the battery problem later.
Common mistakes
Launching experiments but never summarizing learnings
Without a review and repository, hard-won insights evaporate and the org repeatedly relitigates or re-runs ideas it already tested.
Letting proven winners be forgotten
Winning patterns that aren't documented and re-surfaced drop out of new designs, forcing teams to rediscover them or lose the gains entirely.
Is it for you?
Best for
Experimentation leads at organizations running enough experiments that knowledge loss becomes costly
Not ideal for
Very early-stage teams running only a handful of experiments where memory is trivially retained
From the transcript
“one of the mistakes that some company makes is they launch a lot of experiments and never go back and summarize the learnings”
“doing the quarterly meeting of the most surprising experiments”
“one of the things you learned about institutional memories when you have winners make sure to address them and remember them”
From the episode
The ultimate guide to A/B testing
Ronny Kohavi (Airbnb, Microsoft, Amazon)