Build With, Not For: The Product Lab Cohort
Stand up a permanent invite-only user cohort so no fundamental change ever ships cold.
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 93%
Rather than gathering feedback ad hoc and then disappearing to build, Substack maintains a standing invite-only group of around 100 writers on the bleeding edge — the Product Lab. Any change to how Substack fundamentally works goes through mockups, calls, and a live pilot with this cohort before general release. It is the operational expression of 'put users in charge': you cannot claim to hand users control while shipping changes to them unannounced.
Origin
Sachin Monga describes it as one of Substack's operating principles — 'build with writers, build with readers' — which he frames as a sub-principle of putting writers and readers in charge. It was proven on the recommendations rollout and then institutionalized as the Product Lab.
Core principles
- 01Build-with is a sub-principle of user control: shipping changes at people is a form of taking control away.
- 02Feedback rounds are not enough — most teams take the feedback and then build in the dark. The pilot is the point.
- 03A standing cohort is infrastructure. Recruiting one from scratch per feature guarantees you will skip the step under deadline.
- 04Expect day-one shipped scope to look different from the pre-feedback plan. If it doesn't, the process was theatre.
- 05Killing a feature after the pilot is a successful outcome, not a wasted cycle.
How to run it
- 1
Set the default: fundamental changes go through users first
Make it a stated operating principle that anytime you make a fundamental change to how the product works, you bring users along. Strong default, not a case-by-case judgment call.
Pro tip Scope it deliberately to fundamental changes so the process does not choke ordinary iteration.
- 2
Call ten users and show them a mock
Before building, call roughly ten users you think would be interested, mock up what the feature could look like, and walk them through how they would want it to work.
Pro tip Mocking is cheap — Monga's framing is that it is not that hard to just mock up what this could look like. The cost of skipping this is building the wrong thing.
- 3
Run a real pilot, not just a feedback round
The step most teams skip: after collecting good feedback, do not jump straight to build-and-ship. Run a small live pilot with those users and observe actual behaviour.
Watch out Good feedback plus a fast build is how teams convince themselves they were user-led while shipping their original plan unchanged.
- 4
Institutionalize the cohort
Convert the ad hoc group into standing infrastructure: an invite-only lab of ~100 users who want to be on the bleeding edge, ready to be pulled into any feature at short notice.
Pro tip Select for users who are genuinely invested in where the product is going, not just your biggest accounts.
- 5
Let the pilot change or kill the feature
Reshape scope based on what the pilot shows, and be willing to abandon features outright. Lenny reports being in Substack pilots for features that simply did not go anywhere and were dropped.
Pro tip Tell the cohort openly when a feature is being dropped. It builds the trust that makes them keep showing up.
In the wild
Rather than dropping a dialog into every writer's dashboard, Substack called a small group of writers, mocked the feature, took their input on how it should work, and ran a limited pilot with them — Lenny among them — before rolling out.
→ The feature launched in a form materially different from the original concept and became the platform's largest discovery channel, driving over a third of new subscriptions.
The recommendations pilot proved valuable enough that Substack formalized it into an invite-only group of roughly 100 writers, giving the product team a permanent channel for fast feedback on growth tooling.
→ Substack now never rolls a fundamental change out to everyone without passing it through the lab first, and reports that shipped day-one scope routinely differs from the pre-lab plan.
Common mistakes
Stopping at feedback and skipping the pilot
Monga explicitly names this as where most product teams stop: they get good feedback, then just build the thing and ship the thing. The pilot is what surfaces the gap between what users said and what they do.
Recruiting the cohort per-feature
Without standing infrastructure, the build-with step becomes a cost that gets cut under deadline pressure. The lab exists precisely so the step is cheap enough to never skip.
Is it for you?
Best for
Product teams at platforms whose users have livelihoods riding on the product, where a bad unannounced change destroys trust as well as metrics.
Not ideal for
Small internal tweaks and routine iteration, or consumer products where users have no stake and no interest in co-designing.
From the transcript
“we call it build with writers, build with readers.”
“It was like, "Okay, why don't we call up 10 writers who we think might be interested in this? It's not that hard to just…”
“this is just like an invite-only little group of 100 or so writers that we know are interested in being on the bleeding edge of…”
“And often the thing that we end up shipping on day one tends to be pretty different from what we had in mind before we…”
From the episode
Building Substack
Sachin Monga (Substack, Facebook)