The Pilot Program for De-Risking Launches
Build a diverse champion network of real customers to test new features in the wild before you ever launch.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 82%
For a multiplayer product where features can't be tested by one person alone, internal dogfooding stops being predictive as your customer base diversifies. Slack built a standing pilot program of thousands of pre-agreed customers across industries, sizes, and countries, rolling new functionality out to progressively larger real user bases and gathering structured feedback. By launch, the team has real-world evidence from a thousand companies across 50 countries and 20 industries — far more predictive than 'it worked well internally'.
Origin
Described by Noah Weiss as Slack's product development program. He notes it echoes Stripe's practice of picking a couple of motivated customers and building the solution with them.
Core principles
- 01Once your customer base is diverse, internal usage is no longer predictive of external success
- 02Multiplayer features must be rolled out to whole teams or companies, not individuals
- 03Recruit for diversity of industry, company size, and location — and for genuine motivation and risk tolerance
- 04In B2B, a small number of highly motivated customers who can't live without the solution predicts broad demand
How to run it
- 1
Recruit a diverse champion network
Sign up thousands of customers across industries, company sizes, and geographies under agreements that let you roll out net-new functionality to them.
Pro tip Weight recruitment toward diversity of industry, company size, and location so results generalize to your real market.
- 2
Select for motivation and risk tolerance
Pick customers who actively want to be part of the development process and can tolerate rougher, possibly-removed features — champions who love the product enough to endure some pain.
Watch out Not every company wants to beta-test features that might get killed; forcing unmotivated customers in produces noisy, unreliable feedback.
- 3
Roll out progressively to real teams
Because the product is multiplayer, release net-new functionality to whole teams or companies and expand to progressively larger user bases.
- 4
Gather structured feedback at scale
Collect feedback through the product itself and through surveys, so by launch you have confidence from metrics and surveys across many companies, countries, and industries.
Pro tip Apply the Stripe test — if a pilot customer says they 'can't live without it', you've likely built something many other companies will want.
In the wild
Slack maintains a program of thousands of customers on signed agreements, rolling features like Huddles to progressively larger user bases and collecting feedback via Slack Connect and surveys. By the time a big net-new feature launches, it's been used to get real work done across a thousand companies, 50 countries, and 20 industries.
→ Feature teams launch with far higher confidence — grounded in real-world usage rather than non-predictive internal dogfooding — even though a hit is never guaranteed.
Common mistakes
Trusting internal dogfooding as your signal
As the customer base diversifies, 'it worked well internally' stops being predictive because the team no longer resembles the users; only real-world pilots across diverse companies give reliable signal.
Recruiting unmotivated or homogeneous pilot customers
Customers without risk tolerance or motivation won't tolerate rough features and give thin feedback, and a non-diverse pilot pool produces results that don't generalize to the real market.
Is it for you?
Best for
Product teams at a scaling B2B/multiplayer SaaS company shipping net-new features to an increasingly diverse customer base.
Not ideal for
Single-player consumer products testable via simple A/B experiments, or early startups without enough customers to form a diverse panel.
From the transcript
“we have probably thousands of different customers have all signed different agreements now where we can actually roll out to progressively larger user bases because…”
“you want a lot of diversity in terms of industry company size location and so on and I think you want to pick people who…”
“not because it just worked well internally which is no longer that predictive but because it worked well for a thousand different companies 50 different…”
From the episode
The 10 traits of great PMs, how AI will impact your product, and Slack’s product development process
Noah Weiss (Slack, Foursquare, Google)