“you guys are exoptimizely people. So you're very uh versed in experimentation.”
Optimizely
3 recommend/use · 6 sourced episodes
Every sourced reference
Short attributed excerpts only. Timestamps are approximate.
“when I was VP of product ad optimiz Lee”
“ultimately they they acquired me”
“optimizely in its early days were very statistically naive”
“So you need either like Optimizely.”
“category creators like optimizely really paved the way I was an early buyer of optimizely”
“all they're doing is little low risk optimizely a b tests”
Related frameworks
Aha Moment Discovery via Correlation-Then-Experiment
Brainstorm high-value actions, correlate them with conversion and retention, then experiment to prove causation
Building the PLG Team and Org
Start with a growth PM and an analyst, then formalize into a cross-functional PLG org with three growth heads
Clear Is Kind: The Three-Sentence Correction
State the failure, state your belief in them, hand them the responsibility — starting tomorrow
Day One vs Day 100 Feature Packaging
Free the features that create the aha moment; monetize the features whose value only emerges at scale.
Enterprise Price-Ceiling Discovery
Keep asking for more until deals break; losing 20-30% to price is how you find the ceiling.
Evaluative User Research (Hunt for Reasons They Won't Use It)
Test prototypes to find every reason people won't use it — and never let the researcher go alone
False Positive Risk and the Replication Protocol
A p<0.05 win can be 26% likely false when idea success rates are low — replicate borderline results
Feature Team vs. Empowered Product Team Diagnostic
Tell in minutes whether your squad is handed features to ship or problems to solve
Founder-Led Marketing as a Goodwill Bank
Log what you learn, package it per platform, bank goodwill to spend later
Four Leverage Buckets: Where to Start Your PLG Motion
Choose your first PLG investment among acquisition, activation, conversion, and retention by fit and leverage
Incremental OFAT Over Big-Bang Redesign
Decompose big redesigns into one-factor-at-a-time tested steps because ~80% of ideas fail
Institutional Memory of Experiments
Capture wins, losses, and surprises so the org keeps learning instead of relaunching old ideas
Lean Team by Design: Slow Hiring, Player-Coaches, Generalists
Hire painfully slowly, replicate your first-10 DNA, and bet big on the exceptional
Lowercase-c vs Capital-C: AI-Proofing Your Skill Stack
Sort every task into what AI does 80% well and what only a human can do — then invest accordingly
Manufacture the Room (Bending the Universe)
Decide which room you must be in, then engineer a legitimate reason to be standing in it
Match Price to Value with a Value Metric
Pick the unit of value your customer gets, then charge against it so revenue scales as usage grows.
One Click Faster (Clock Speed Management)
Three tactics to raise an organization's pace without lowering its bar
PLG Is Fundamentally DLG (Data-Led Growth)
Giving away a free product only pays off if you capture and analyze its usage data
Price Your Product Like Your Roadmap
Treat pricing as a living system revisited every 6-12 months, not a set-and-forget decision.
Reactive Pricing After Demand Signal
Let users pull pricing out of you, then validate willingness-to-pay and margins
Retention as Habit-Building Plus Expansion
Retain by baking high-frequency habits into the product, then convert steady usage into expansion revenue
Sample Ratio Mismatch (SRM) Validity Check
Verify your traffic split matches its design before trusting any experiment result
Solution-Weighted Discovery
Spend a little time validating the problem; spend most of it winning on the solution
The 200K-User Experimentation Readiness Rule
Below ~200K users detect only big effects; use the wait to build platform and culture
The Best-Teams Adoption Heuristic
Adopt a practice only if several of the best teams already use it — then strip the culture from the technique
The Cross-Functional Pricing Process
Stand up a pricing committee, mine demand and willingness-to-pay with structured surveys, then set price.
The Diseases of a Devolving Product Company
Four diagnosable diseases that turn a great product company into a feature factory
The Durable Wrapper Test
Own a workflow you care about for a decade, then orchestrate 20 models beneath it
The First 30 Seconds Onboarding Bet
Earn the next 30 seconds by making one thing feel magical fast
The Five PLG Readiness Components
The five things a product must have before a product-led motion can work
The Four Product Risks and Their Owners
Valuable, usable, feasible, viable — any one fails and the product fails
The Full PLG Funnel Audit
Walk your own funnel as a confused buyer, then overlay the data to find your highest-leverage fix
The Micro-Influencer Wide-Net Playbook
Skip the big creators; onboard thousands of trusted micro-influencers like teammates
The One-Quarter Empowered Team Experiment
Convert a single feature team into an empowered product team in one low-risk quarter
The Org Chart With Your Name On It
Win the promotion by solving your boss's org problem, not by asking for your next title
The Overall Evaluation Criterion (OEC)
Define one success metric with countervailing guardrails that predicts long-term lifetime value
The PLG Data & Infrastructure Stack
The three infrastructure layers plus per-funnel-stage tools every PLG motion needs
The Product Manager's Four Areas of Homework
The four knowledge domains a PM must own because nobody else on the team can bring them
The Same-Day Prototype Validation Loop
Idea in the morning, real users testing by afternoon, decision by evening
The Technology Wave Specialization Ladder
Ratchet down on a new technology while it's still weird — teardown, no-code, then fun build
The Three Sacred Accesses
Delegate anything off the PM's plate except direct access to customers, engineers, and stakeholders
The Word-of-Mouth Machine PMF Test
Real product-market fit is organic pull, not award-winning launch spikes
The Zone of Genius Calendar Audit
Bucket last quarter's calendar by joy, then ask what only you can do
Twyman's Law: Investigate the Too-Good-to-Be-True Result
Any surprising or extreme experiment result is probably wrong until you've hunted for the flaw
People in these episodes
Related resources
- Acorns
- Ahrefs
- Airbnb
- Airtable
- AKG Media
- Amazon
- Amazon Prime
- Amazon Redshift
- Amplitude
- Appcues
- Apple
- Apple Podcasts
- Bing
- Blink cameras
- Bluey
- Build
- Calendly
- Canva
- Census
- ChatGPT
- ChatPRD
- Chernobyl
- Circe
- Clearbit
- Conjoint analysis
- Continuous Discovery Habits
- dbt
- Dovetail
- Dropbox
- eBay
- Empowered
- Endgame
- Eppo
- Evernote
- Figma
- First Collab
- Flatfile
- Gamma
- Gas
- GitHub
- GitLab
- GoodUI
- Google Analytics
- Google Docs
- Google Slides
- GPT Store
- Hard Facts, Dangerous Half-Truths, and Total Nonsense
- High Growth Handbook
- Hightouch
- Hotmail
- HubSpot
- IDEO
- Insight Out
- Inspired
- Intercom
- iPhone
- iPod
- Jira
- LaunchDarkly
- Lenny's Podcast
- Lenny's Podcast website
- Linear
- Loom
- Maven
- Menlo Ventures website
- Metronome
- Microsoft
- Midjourney
- Miro
- Mistakes Were Made (But Not by Me)
- Mixpanel
- Modern Treasury
- Monetizing Innovation
- MSN
- Mythic Quest
- Netflix
- Netscape
- No Rules Rules
- Nobody Wants to Read Your Sh*t
- Notion
- OpenAI API
- OpenView
- Orb
- Pace
- Pacifica
- Pendo
- Perplexity
- Pocus
- Poor Things
- PostHog
- PowerPoint
- Product Hunt
- ProfitWell
- Reforge
- Ronny Kohavi’s live cohort-based experimentation course
- Round
- SAFe
- Salesforce
- San Francisco
- SAP
- Scaling People
- Scrum
- Segment
- Shoe Dog
- Shopify
- Silicon Valley
- Silicon Valley Product Group
- Simon-Kucher
- Slack
- Smith & Diction
- Snowflake
- Someday
- Spotify
- Sprint
- Stanford d.school
- StoryBots
- Stripe
- The Almanack of Naval Ravikant
- The Lazarus Project
- The Lost Interview
- The Mom Test
- The Odyssey
- The Three-Body Problem
- The Wandering Earth II
- TikTok
- Toplyne
- Trustworthy Online Controlled Experiments
- Userpilot
- UserTesting
- Van Westendorp Price Sensitivity Meter
- Vanta
- Voicepanel
- Waymo
- Whimsical
- Windows Indexer
- Working Backwards
- X (Twitter)
- Xero
- YouTube
- Zoom
- ZoomInfo
- Zuora
Spot an error or want this page removed? Request a correction or removal.