LLenny's Podcast
← All episodes
Archie Abrams (VP Product, Head of Growth at Shopify)07 November 2024

Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, prioritizes intuition, and builds toward a 100-year vision

7Frameworks
14Insights

Frameworks in this episode

Leadership3 steps

Accountability by Metrics OR Taste

Building without KPIs works only with an exceptionally opinionated founder/taste-holder; otherwise you need metrics.

Read →
Sales3 steps

Discount to Signal Value and Serve the Emotional Job

Set a high list price to signal quality, then discount hard with urgency to let buyers complete an emotional job.

Read →
Innovation4 steps

Long-Term Holdout Experimentation

Ship experiment winners fast, but keep the exposed cohort held and re-check GMV impact at 3, 6, and 12 months.

Read →
Strategy3 steps

Optimize Absolute Throughput, Not Local Conversion Rates

Goal teams on the total number of people who complete a funnel stage, never the conversion rate.

Read →
Marketing3 steps

Reduce Monetary Friction to Unlock Hidden Winners

Lower early cost barriers to causally give marginal customers time to succeed, revealing valuable users you'd otherwise lose.

Read →
Innovation3 steps

Ship Neutral Experiments on Intuition (Aim-Heavy)

When a test comes back neutral, ship the version you'd have built from a blank slate, not the incumbent control.

Read →
Strategy3 steps

The Portfolio Growth Model: Optimize for Churn

Lower the barrier to entry and accept high churn; a few power-law winners make the whole cohort pay off.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster17:30

A Third of Your 'Growth Wins' Have No Long-Term Impact

Archie estimates 30-40% of experiments that show a clear short-term lift produce no incremental long-term GMV. In many cases the growth team simply pulled forward success that would have happened anyway, or captured a marginal user who wasn't worth much. It's brutal news for growth teams, but knowing it is how you actually build a business that keeps growing.

  • Roughly 30-40% of short-term wins show zero long-term GMV lift
  • Often it's a pull-forward effect, not genuine incremental value
  • Sometimes the incremental user acquired simply isn't worth much
  • Painful to hear, but essential for building durable growth

“it's probably in the it's it's in the 30 to 40% range”

Archie Abrams · 17:30

“unfortunately I think that's brutal probably more common than we like to we like to believe”

Archie Abrams · 18:00
#experimentation#growth#pull-forward#measurement#myth

Hot Take· 5

Hot Take06:30

Why Shopify Optimizes for Churn Instead of Retention

Unlike most SaaS companies obsessed with retention, Shopify deliberately lowers the barrier to entry knowing most merchants will fail. The model works like angel investing: a small number of breakout merchants (the Alberts and Figs of the world) make an entire cohort profitable, so the goal is getting as many people as possible to try entrepreneurship.

  • Most businesses fail, and Shopify accepts that as the cost of getting more people in the door
  • A few outlier merchants who become huge make the whole cohort successful
  • The mindset mirrors angel investing: a couple of winners carry the portfolio
  • It is not that they want people to leave, they just don't optimize every new shop for sticking around

“most of Angel Investments are not going to work out but the couple that do make that entire investment uh kind a portfolio successful”

Archie Abrams · 09:30
#growth#churn#retention#shopify#business-model
Hot Take25:00

Why Shopify Goals Teams on Absolute Numbers, Not Conversion Rates

When teams optimize local conversion rates, the easiest cheat is to constrict the funnel step right before them so only higher-intent users get through. Shopify avoids this by goaling teams on the absolute number of merchants who reach the end of their part of the journey, which incentivizes getting more people in the door rather than gaming a rate.

  • Optimizing a conversion rate incentivizes making the previous step harder
  • Fewer, higher-intent users inflate your rate but shrink the total
  • Goaling on absolute numbers pushes teams to widen the funnel, not constrict it
  • A lower conversion rate can be worth it if CAC drops more than LTV

“the easiest way to increase retention is always to constrict the funnel stage one above the retention metric you're trying to optimize for the simplest…”

Archie Abrams · 25:00

“if I make it harder to sign up it's going to be very easy to increase sign up to activated rate because I just have…”

Archie Abrams · 28:00
#metrics#conversion-rate#funnel#incentives#growth
Hot Take41:00

Why Shopify Ships Experiments That Test Neutral

Because Shopify is 'aim heavy' and trusts intuition, a neutral experiment result doesn't mean revert to control. If neither variant is measurably better, they ask what they'd build from a blank slate and ship the version they believe is right for merchants. A pre-filled online-store editor tested neutral on conversion but months later drove a massive GMV lift.

  • A neutral result means neither option is better, so pick the one you believe in
  • The 'aim heavy' culture defaults to intuition when data is inconclusive
  • Pre-filled store sections had no conversion lift but big long-term GMV impact
  • Better stores converted more visitors, giving merchants early momentum

“if the intuition is right that this probably is helping Merchants why do we start with that the original control was better if it's neutral…”

Archie Abrams · 41:30
#experimentation#intuition#aim-heavy#shipping#growth
Hot Take44:30

Why the 'How' Determines Strategy More Than the 'What'

A lesson from Toby: in a technology company, the technical architecture (the 'how') shapes strategy even more than what you build or who you build it for. Every six weeks leaders review every project with Toby, often diving into technical detail like whether to build a CSV importer in the core codebase or as a separate app, because those choices preserve long-term optionality.

  • Technical architecture drives strategy more than the what and who
  • Leaders review every R&D project with Toby every six weeks
  • A 30-minute discussion once centered on how to build CSV importers
  • Getting the 'how' right sometimes means shipping slower or passing on deals

“the how the technical architecture determines strategy in a technology company even more than the kind of what and who we're building for”

Archie Abrams · 45:00
#architecture#strategy#engineering#toby#optionality
Hot Take48:00

Shopify Bans KPIs and OKRs in Its Core Product

In Shopify's core product org, KPIs and OKRs are essentially banned. Instead of holding teams accountable to a metric over six months, decisions rest on conviction and taste about what's the right thing to build for the future of commerce. Teams still use data as one input, but it isn't the overriding driver, which lets them take more risk, at the cost of some subjectivity.

  • KPIs and OKRs are banned in core product
  • Decisions rest on conviction that it's the right technical foundation, not a metric
  • Data is one lens among several, alongside qualitative input and product sense
  • The upside is bolder, higher-quality shipping; the downside is subjectivity and squishiness

“we rant against kpis are basically banned as a and okrs are banned and and all that”

Archie Abrams · 48:30

“kind of taste is kind of what drives a lot of what we're shipping in core”

Archie Abrams · 50:00
#kpis#okrs#taste#product-management#shopify

Explainer· 4

Explainer08:30

How Payments Revenue Lets Shopify Tolerate Churn

Shopify can afford not to obsess over retention because most of its revenue comes from payments tied to GMV, not subscriptions. As a merchant grows, Shopify grows with them, so the winners generate outsized net revenue that covers everyone who never succeeds. A typical SaaS company monetizing at $29/month has no such upside.

  • Most SaaS companies only monetize via a flat subscription, capping upside per customer
  • Shopify monetizes GMV through payments and services, so revenue scales with merchant success
  • Net dollar retention on the winners is enormous and offsets the failures
  • This monetization model, more than cheap acquisition, is why churn matters less

“as folks get bigger because we're monetizing on that gmv that that Merchant is producing the revenue the merchant is producing in the form of…”

Archie Abrams · 09:00
#monetization#payments#gmv#ndr#saas
Explainer11:30

Why Shopify Re-Checks Every Experiment a Year Later

Shopify holds out cohorts and revisits experiments at 3, 6, 12 months and beyond to see the true long-term GMV effect, not just the short-term lift. The experimentation tool automatically pings everyone involved with updated results so teams can't hide from what actually happened. This long horizon lets them hold themselves accountable to the metric they truly care about.

  • They keep long-term holdouts and revisit the GMV curve a year or more later
  • Two layers of holdouts: 5% across all quarterly changes, plus 50/50 splits for new-merchant changes
  • The winner ships to 100%, but the original cohort is tracked for the long-term read
  • Experimenters get automatic pings at 3, 6, and 12 months with updated results

“I would encourage everyone if you can look at some of the experiments that you thought were your biggest winners look at the down tree…”

Archie Abrams · 14:00

“a more short-term metric number of people who become a paying shop or number of people make their first sale in Shopify and then you…”

Archie Abrams · 15:00
#experimentation#holdouts#gmv#measurement#growth
Explainer16:00

How Reducing 'Monetary Friction' Unlocks Hidden Winners

Conventional wisdom says discounts attract low-quality customers, but Shopify found the opposite. Lowering monetary friction (trial terms, incentives, price) causally gives struggling founders more time to make their business work, unlocking a class of valuable entrepreneurs who would otherwise have given up. This is one of the pockets of high-value merchants that only shows up in long-term measurement.

  • Monetary friction covers trial dynamics, incentives, app-store credits, and price point
  • Common belief: discounts bring in lower-quality users, so avoid them
  • Reality: a small monetary boost causally extends a founder's runway to succeed
  • It unlocks merchants who would have quit without that reduced friction

“if I give you a little less a little monetary boost and reduce that monetary friction I can actually causally change your ability to become…”

Archie Abrams · 17:00
#pricing#monetary-friction#discounts#activation#growth
Explainer42:00

Building Toward a 100-Year Vision Instead of This Quarter

Shopify's decisions are oriented around being a 100-year company, driven by CEO Toby. That means resisting the seductive short-term revenue of onboarding big legacy brands, because most of today's giants will be gone in decades. The real long-term bet is helping every new business start and grow, encoded in the company principles.

  • The company optimizes for the long-term success of merchants over quarterly wins
  • Chasing big-brand deals feels good short-term but doesn't build a 100-year company
  • Most of today's large brands will be out of business in 30-50 years
  • Core principles: make the best product, make money to do more of it, never reverse the order

“if you're thinking about the longterm 100 years from now guess what all the big brands of today be out of business in many of…”

Archie Abrams · 43:30

“make the best product in the world make money to do more of one never reverse principles two and three”

Archie Abrams · 42:30
#vision#long-term#strategy#shopify#principles

Story· 2

Story38:00

The Payment-Failure Win That Turned Out to Be Nothing

Shopify ran a classic growth play: aggressively alerting merchants when their payment failed (dunning). It produced major short-term lift, but a year later there was no long-term GMV impact. The reason was selection bias: merchants letting a payment lapse weren't really committed, so recovering them didn't create real entrepreneurs.

  • Dunning alerts for failed payments drove strong short-term recovery
  • Six to twelve months out there was no long-term GMV lift
  • Selection bias: merchants who let payments fail weren't truly committed
  • A textbook example of pulling forward revenue that wasn't durable

“the people who are letting that that payment fail probably weren't actually that dedicated to this entrepreneurship craft they may have updated their credit card…”

Archie Abrams · 39:00
#dunning#story#selection-bias#experimentation#growth
Story1:08:30

The Real Power of Discounting, Learned at Udemy

At Udemy, discounting did two things: a high list price signaled quality in a murky new category, and deep discounts (90-99% off) made courses affordable. But the deeper insight was emotional: education is aspirational, and the act of purchasing a course itself feels like progress. That emotional job, triggered by urgency and discounts, drove strong repeat purchasing and retention.

  • A high list price signals quality where value is unclear (like early online courses)
  • Deep discounts (90-99% off) made the aspirational purchase affordable
  • The emotional job-to-be-done is feeling progress, satisfied by the act of buying
  • Discount plus urgency let people repeat that emotional purchase, driving retention

“there's an emotional job that's even more important which is I'm feeling like I'm making progress in my educational journey and just the act of…”

Archie Abrams · 1:10:30
#discounting#udemy#pricing#psychology#retention

Takeaway· 2

Takeaway18:30

Advice for Teams Who Can't Run Year-Long Holdouts

For teams without the luxury of long holdouts, Archie's rule is simple: if a change shows short-term lift, ship it, because no short-term movement means no long-term lift either. Just don't overestimate the impact. Instrument the earliest real signs of success and look as far down the funnel as you can, but don't wait forever.

  • If it moves the short-term metric, ship it; it probably won't hurt
  • No short-term impact means no long-term impact, so don't agonize over waiting
  • Instrument the earliest genuine signs of product success
  • Measure as deep down the funnel as you can, but stay reasonable about impact

“if you didn't move the upper the short-term impact you're not going to have the long-term lift so still ship if it's short-term lift just…”

Archie Abrams · 19:30
#experimentation#growth#advice#measurement#shipping
Takeaway59:30

You Need Either Metrics or a Founder With Extreme Taste

Archie's honest caveat for anyone inspired to ditch metrics: it only works with an extremely opinionated founder or small group who define what good looks like. Otherwise you get teams building whatever they want with no accountability, which is a very bad state. Metrics are the common accountability mechanism; strong founder taste is the rare alternative, and you need one of the two.

  • Two ways to drive accountability: metrics, or a founder/small group with extreme taste
  • Without either, teams build haphazardly with no accountability
  • Shopify has drifted into the no-accountability state before and it was very bad
  • Strong founder taste (Toby, Brian Chesky, Elon) is rare, so most companies should use metrics

“either have to use my sense is metrics as accountability which is the most common kind of way to drive accountability and focus or extremely…”

Archie Abrams · 1:00:00
#taste#accountability#founders#metrics#leadership