Artificial Density Testing
Manually cram enough users into one cluster to get a clean yes/no on a social product
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 93%
To test a network-effects product you must remove the ambiguity of 'did it fail or did it just lack enough users?' Bier manually seeds an entire tight community (like one high school) to full density at once, so every user has enough friends to reach the aha moment. The goal is signal, not scale, so you never want to walk away unsure whether the execution or the idea was at fault.
Origin
Nikita Bier's testing method (distinct from his growth method), based on a leaked BuzzFeed memo about how his team seeded schools; he stresses this is how they tested apps, not how they grew them.
Core principles
- 01A reproducible testing process influences success probability more than anything else
- 02Eliminate every confounding variable so a test gives real signal
- 03Seed the ideal version manually even if it doesn't scale, to learn whether people care
- 04Density and synchronous adoption are prerequisites for a social flywheel to even start
- 05Live chat support is the best channel for user research and eliminating the 'were they treated well?' variable
How to run it
- 1
Pick one tight, self-identifying community
Choose a single dense cluster (e.g. a high school where students put initials like 'RHS' in their bios) small enough to saturate.
Pro tip Launch where the calendar helps, Bier chose the US school with the earliest start date to move fast.
- 2
Saturate it with marketing on every channel
Run targeted ads and a dedicated Instagram account that follows students so they see the app roughly three times and adopt synchronously.
Pro tip People need to see a marketing message about three times before they'll download.
Watch out This is a testing tactic, not a growth strategy; you cannot follow every kid school by school to scale, and copying it as growth fails.
- 3
Get the first ~100 users on at once
Drive enough synchronous adoption that each user has real friends present, so they can actually reach the aha moment.
- 4
Add live chat to harvest signal
Put 24/7 live chat support in the app to give a white-glove experience and, more importantly, mine users' problems and feature ideas.
Pro tip Route interesting feedback into Slack and turn the best ideas into features.
- 5
Read the binary result
With confounders removed, judge with conviction whether the product has legs; if it works at the single-cluster level you can relaunch it anytime.
Watch out Never leave a test able to be explained away by 'maybe the execution was bad'; that wastes the team's mobilization energy.
In the wild
Bier's team saturated one high school with ads and a dedicated Instagram account that followed and follow-backed students, getting the whole school to adopt at once so they could see if the app had legs without wondering whether users had enough friends.
→ 40% of the school downloaded TBH in the first 24 hours and it spread to neighbouring schools, confirming the product worked before national rollout.
Common mistakes
Confusing the test tactic with the growth engine
Founders DM Bier that they replicated the school-seeding at 15 schools and it stopped working; that manual method only exists to test, the app must grow itself afterward.
Leaving confounders in the test
If you don't guarantee density and a good experience, a flat result could mean too few friends, not a bad idea, and you learn nothing.
Is it for you?
Best for
Founders testing a network-effects or social product that needs friend density to deliver value
Not ideal for
Single-player or utility products where one user gets value without others present
From the transcript
“develop a reproducible testing process and that will actually influence the probability of your success more than anything”
“This is not the way we grew the app. This is how we tested apps.”
“put live chat customer support in your app like like 24 hours a day”
“figure out a way to eliminate all those potentially confounding variables uh so you can know immediately whether something's working or not”
From the episode
How to consistently go viral: Nikita Bier’s playbook for winning at consumer apps (co-founder of TBH, Gas, advisor, investor)