LLenny's Podcast
← All people
Verified guest

Ronny Kohavi

Rani is seen by many as the world expert on a b testing and experimentation

1 episode · 1 sourced appearance

Appearances and evidence

Short attributed excerpts only. Timestamps are approximate.

Frameworks in these episodes

Episode-linked context, not an unsupported claim of authorship.

Mindset4 steps

False Positive Risk and the Replication Protocol

A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results

Read →
Innovation4 steps

Incremental OFAT Over Big-Bang Redesign

Decompose big redesigns into one-factor-at-a-time tested steps because ~80% of ideas fail

Read →
Leadership4 steps

Institutional Memory of Experiments

Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas

Read →
Strategy4 steps

Sample Ratio Mismatch (SRM) Validity Check

Verify your traffic split matches its design before trusting any experiment result

Read →
Strategy4 steps

The 200K-User Experimentation Readiness Rule

Below ~200K users detect only big effects; use the wait to build platform and culture

Read →
Strategy4 steps

The Overall Evaluation Criterion (OEC)

Define one success metric with countervailing guardrails that predicts long-term lifetime value

Read →
Mindset4 steps

Twyman's Law: Investigate the Too-Good-to-Be-True Result

Any surprising or extreme experiment result is probably wrong until you've hunted for the flaw

Read →

Resources in their episodes

Related guests

Spot an error or want this profile removed? Request a correction or removal.