Aha Moment Discovery via Correlation-Then-Experiment
Brainstorm high-value actions, correlate them with conversion and retention, then experiment to prove causation
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 94%
Qu's method for defining a product's activation metric: brainstorm candidate high-value actions, run correlation analysis measuring how each lifts both conversion and retention versus average, shortlist the strongest, and then launch experiments to drive those actions and confirm causation. At GitLab this produced 'two users using two features in the first 14 days.' The key discipline is looking at conversion AND retention together, and never mistaking correlation for causation.
Origin
Developed by Hila Qu's growth team at GitLab; the correlation-then-experiment structure echoes Facebook's classic '10 friends in 7 days' aha-moment analysis, which Qu explicitly references as the template.
Core principles
- 01An aha moment is the first time a user experiences the product's value
- 02Look at conversion AND retention together — either alone gives an incomplete picture
- 03Correlation identifies candidates; only experiments prove causation
- 04A multi-workflow product may need a combination of actions rather than one
- 05The window must be reasonably quick but realistic for your product's complexity
How to run it
- 1
Brainstorm candidate high-value actions
Have the growth team list the actions or behaviors that plausibly indicate a user is getting value (e.g. at GitLab: merge first PR, run first pipeline).
- 2
Run correlation analysis on each candidate
For each high-value action, measure whether users who did it in their first N days show a higher 90-day conversion rate and 30-day retention rate versus the average.
Pro tip Look at both conversion and retention — sometimes one alone doesn't give the full picture.
- 3
Shortlist the strongest lifters
Compare candidates against the average; the actions that lift conversion and retention the most are your aha-moment candidates. Pick a single action if one clearly stands out, or a combination if many workflows drive value.
Pro tip GitLab combined actions because teams come for different reasons (security, CI/CD) — 'any two of the high-value features' captured the platform's breadth.
- 4
Experiment to validate causation
Launch experiments that get more users to perform the candidate actions and check whether conversion and retention actually rise.
Watch out Data only isolates correlation — people who did the action were more likely to convert doesn't mean forcing the action causes conversion. Experiments are where you finally validate.
In the wild
GitLab brainstormed ~10 high-value actions, ran correlation analysis on 90-day conversion and 30-day retention, and found no single action dominated because teams use the platform for different reasons. They combined signals into two users using two or more features within the first 14 days — two users captures the team/collaboration value, two features captures the platform value, and 14 days is quick but realistic for a complex product.
→ A defensible, data-grounded activation metric tailored to a multi-workflow platform rather than a copied single-action milestone.
Common mistakes
Treating correlation as causation
Finding that users who take an action convert more does not prove that driving the action causes conversion. Skipping the experiment step means optimizing toward a metric that may not move outcomes.
Looking at only conversion or only retention
Each metric alone gives an incomplete picture of value. An action can lift one without the other; you need both to identify a true aha moment.
Is it for you?
Best for
Growth PMs and analysts at PLG products trying to define or validate an activation/aha-moment metric
Not ideal for
Very early products with too little usage data to run meaningful correlation analysis
From the transcript
“We ended up have something along the line of two users, two features used in the first 14 days”
“we did a correlation analysis to understand hey, those are the 10 high value actions we believe”
“What's the 30-day retention rate? Because we look at both. Sometimes um you you you only look at retention or you only look at conversion,…”
“in data you are only isolating correlation. You are not proving causation”
“at experimentation is the step you will finally kind of validate that”
From the episode
The ultimate guide to adding a PLG motion
Hila Qu (Reforge, GitLab)