LLenny's Podcast
← All episodes
Crystal Widjaja, Gojek and Kumu31 July 2022

How to scrappily hire for, measure, and unlock growth

7Frameworks
14Insights

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 1

Hot Take17:00

You Can Run Real Experiments With Just 30 People

Crystal pushes back on the idea that startups are too small to experiment. More data increases the precision of your trends, not the direction of them, so early-stage teams still learn the underlying signal. Her framing: what's better than 30 data points is 100, but what's better than zero is definitely 30 and cheap-to-run tests deliver disproportionately at that scale.

  • More data improves precision, not the underlying trend direction.
  • With a small sample you get the same trends, just less precisely.
  • Look for a large proportion (e.g. ~20 of 30) doing the target behavior.
  • Cheap non-scaling tests deliver more per experiment at 30 people than at 100.

so what's better than having 30 data points certainly having 100 but what's better than having zero is definitely 30.

Crystal Widjaja · 18:00

even if you have a sample size of the data you get back generally does not change but its precision will

Crystal Widjaja · 17:30
#experimentation#statistics#early-stage#growth

Explainer· 4

Explainer11:30

Why a Super App Hasn't Emerged in the US

Crystal explains why the super-app model dominates Southeast Asia but not the US. Consumers there grew up trusting integrated conglomerates, whereas Americans are wary of any one company knowing too much. She adds a hardware angle: many users leapfrogged straight to phones with limited storage, so when the phone fills up they delete apps rather than photos, forcing consolidation into one app.

  • Southeast Asia has cultural trust in conglomerates that own malls, housing, and schools.
  • Americans shy away from one company knowing too much (e.g. "does Google know too much about me").
  • Many Asian users leapfrogged to phones without owning a computer.
  • Limited device storage forces users to delete apps, so surviving apps must be super apps.

there's a sense of trust in a conglomerate whereas in america we already kind of shy away from like does google know too much about…

Crystal Widjaja · 11:30

when your phone is full are you going to delete a photo of your kid or are you going to delete this app you're probably…

Crystal Widjaja · 12:00
#super-app#strategy#southeast-asia#market
Explainer40:00

Treat Metrics as News, Not Entertainment

Crystal's core diagnosis of why analytics efforts fail: people track data to watch their OKRs go up or down, but never use it to find insights. She likens it to using Twitter for "news" when you're really using it for entertainment consuming interesting facts without acting on them. Real news is information that changes what you do in the real world; if nothing changes, you were just entertained.

  • Most teams track data to monitor OKRs, not to identify actionable insights.
  • Consuming metrics passively is entertainment, not news.
  • Real news is information that changes what you do in the real world.
  • If a data point doesn't change your behavior, it delivered no value.

real news is information that changes what you do in the real world

Crystal Widjaja · 41:00

do not treat metric gathering as entertainment like it's not there for you to be like oh that's interesting how novel and then not act…

Crystal Widjaja · 41:00
#analytics#metrics#instrumentation#insights
Explainer41:30

Measurements Aren't Insights: The Girlfriend's Cousin Analogy

Crystal distinguishes measurements from insights. A measurement is a raw observation ("power users do 4x more bookings") that your database obviously confirms but doesn't tell you why or what to do. Her analogy: seeing your girlfriend with an unknown man is an observed fact; the hypothesis is cheating, but the insight might be that he's her cousin and you're paranoid. Insight arrives once you answer the "why" and can act differently, like giving power users a free-shipping discount only on high-GMV baskets.

  • A measurement is an observation; an insight answers why and tells you how to act.
  • "Power users do 4x more bookings" is a fact, not an insight, without context.
  • The girlfriend-and-cousin analogy shows how the same observation yields different insights.
  • Real insight comes from instrumenting event properties to segment behavior, then testing a hypothesis.

measurements do not equate to insights a measurement would be an observation it's a data point in your database

Crystal Widjaja · 41:30

your hypothesis could be that your girlfriend is cheating on you but the insight uh the actual fact might be that she's not cheating on…

Crystal Widjaja · 42:00
#analytics#insights#instrumentation#segmentation
Explainer44:30

The Telltale Sign of Bad Event Instrumentation

Crystal can spot weak analytics just from the instrumentation spec: lots of events with only one property or none. Using a Gojek "map loaded" event, she shows the properties that matter number of drivers visible, pickup city, lat/long, surge pricing, minimum fare, voucher code. Those let you learn that a user seeing two drivers converts far less than one seeing five, then dig into where and why. Without that context, you can't explain non-conversion.

  • Bad tracking looks like many event rows each carrying one property or none.
  • A "map loaded" event should capture drivers visible, city, lat/long, surge, fare, voucher.
  • Rich properties reveal that seeing 2 drivers converts worse than seeing 5.
  • Missing context makes it impossible to ask or answer why a user didn't convert.

the symptom of a bad data tracking approach is you have a ton of rows with a ton of events but every event has like…

Crystal Widjaja · 44:30

when a user only sees two drivers on the screen they're much less likely to convert than a user who sees five drivers on a…

Crystal Widjaja · 45:30
#instrumentation#event-tracking#analytics#data

Story· 6

Story00:00

Renting a Stadium to Hire 60,000 Gojek Drivers in Weeks

Crystal recounts one of Gojek's most audacious early growth moves: renting an actual stadium, staffing long lines with boxes of pre-loaded phones and SIM cards, and onboarding tens of thousands of motorcycle-taxi drivers in a couple of weeks. She admits the risk of joining felt huge when she arrived to find the company operating out of a house, but the product was already growing fast at ~4,000 orders a day.

  • Gojek ran mass driver hiring by literally renting a stadium (a couple of football fields).
  • Roughly 60,000 drivers were onboarded in a couple of weeks.
  • Setup was long lines, boxes of phones and SIM cards handed to drivers.
  • Company was still tiny (in a house) but already growing at ~4,000 orders/day.

we ended up renting a stadium to just hire like 60 000 drivers in a couple of weeks

Crystal Widjaja · 00:00

when i got there it was in a house and i realized i've probably made a huge mistake

Crystal Widjaja · 00:00
#growth#hiring#gojek#scrappy#operations
Story13:00

Wizard-of-Oz Tests: Validating Features Without Building Them

Crystal describes Gojek's scrappy playbook for testing ideas before writing code. To test a subscription feature, they added 100 drivers to a WhatsApp group, had them sell the package on rides, and manually processed vouchers in the backend. For a new onboarding screen, a designer overlaid a mockup on a screenshot and shipped it as an in-app message. The point: manifest the desired user experience as fast as possible, then build only what works.

  • Tested a subscription feature via a WhatsApp group of 100 drivers selling manually, with interns processing vouchers in the backend.
  • Faked a new onboarding screen by overlaying a designer mockup on a screenshot and sending it as an in-app message.
  • Used Typeform surveys and quizzes as fake features to gauge demand.
  • Built in-app web deployment so web-testable features didn't wait on mobile app releases.

this concept of doing things that are somewhat crazy but validate a point doing stuff that don't scale especially i think is really the bread…

Crystal Widjaja · 13:30

it's really this wizard of oz experience we don't have to build anything

Crystal Widjaja · 14:30
#experimentation#scrappy#wizard-of-oz#product#gojek
Story19:00

Hacking Trust: Using Friends' Orders to Sell New Restaurants

For GoFood, Crystal's team realized users needed trust before ordering from an unfamiliar merchant. Using existing Facebook Connect permissions, they surfaced food that a user's friends had purchased and liked. Users shown a friend's purchase were twice as likely to order from a brand-new restaurant lifting GMV by solving for trust rather than optimizing conversion directly.

  • Diagnosed that trust, not conversion mechanics, was blocking orders from new merchants.
  • Leveraged existing Facebook Connect permissions to see friends' past purchases.
  • Showed users food their friends had bought and liked as social proof.
  • Users with the feature were 2x more likely to order from a brand-new restaurant, raising GMV.

we actually looked at the food that their friends had purchased and used that as a data set of hey here's food that lenny purchased…

Crystal Widjaja · 19:30

you would be twice as likely to purchase from a brand new restaurant than users who did not have this feature

Crystal Widjaja · 19:30
#growth#trust#social-proof#gofood#conversion
Story32:00

Turning Drivers Into GoPay Salespeople With a Captive Audience

To drive GoPay adoption, Crystal's team built a small service that detected when a driver was allocated a customer with no digital balance, then messaged the driver an incentive to get the customer to top up with cash. Trapped together in the car, drivers became remarkably effective salespeople and customers felt the benefit explained directly. It became ~60% of acquisition with no change to the product's physics, just a new use of an existing lever.

  • A backend service checked whether an allocated customer had ever used GoPay.
  • If not, it messaged the driver an incentive to convert them via cash top-up.
  • Drivers' captive attention in the car made them highly effective salespeople.
  • Drove ~60% of GoPay acquisition once released, with no change to product physics.

you wouldn't believe how great of a salesperson someone can be when you were literally trapped in a car with them going somewhere

Crystal Widjaja · 33:00

it was huge it was like 60 of acquisition once we released that

Crystal Widjaja · 33:30
#growth#gopay#incentives#distribution#gojek
Story38:00

Beat Subscription Churn With a Pause Button, Not a Cancel

Advising a beer D2C brand, Crystal found the top cancellation reason was "I still have too much beer." The app let users cancel or resume but not pause a permanent solution to a temporary problem. Adding a pause button alleviated churn that was otherwise hard and expensive to reacquire. Lenny notes the same pattern powered one of Airbnb's biggest wins: a snooze/pause feature for listings.

  • Top cancellation reason for a beer subscription was simply having too much beer.
  • App only allowed cancel or resume, forcing a permanent fix for a temporary problem.
  • Adding a pause button solved churn at the exact constraint where users dropped off.
  • Airbnb's listing snooze/pause was one of its biggest wins for the same reason.

canceling the subscription is a permanent solution to having too much beer how do you make a temporary solution that solves the actual problem

Crystal Widjaja · 38:30

this is actually one of the biggest wins is adding a snooze feature to your listing

Lenny Rachitsky · 39:30
#churn#retention#subscriptions#product#d2c
Story53:30

Hiring Growth: A Take-Home Experiment-Design Case Study

To hire growth people, Crystal screens for first-principles bias with a case study: given a claimed result, how do you know it's true, and how would you design an experiment to test it? She wants candidates who sample randomly and take a measured approach, not those who assume their feature will work. She gives a take-home (about four hours over five days), not a live test, and specifically values candidates who admit they had to Google it and read white papers.

  • Screens for first-principles bias: "how do you know this is true?" plus an experiment design.
  • Wants random sampling and a measured, deliberate approach over assuming a feature will work.
  • Uses a take-home (~4 hours over 5 days), not a live test, to see real quality.
  • Values candidates who admit they Googled it and read white papers to figure it out.

i actually look for that first principle bias so i'll give people case studies of like here's what we see how do you know that…

Crystal Widjaja · 54:00

we like to hear people say that they literally had to google this and read a bunch of white papers

Crystal Widjaja · 56:30
#hiring#growth-team#interviewing#experiment-design

Tool· 1

Tool47:00

Crystal's Recommended Analytics Stack by Company Stage

Crystal gives a stage-based tooling recommendation. Very early with a single data warehouse: Google Data Studio (free), or Metabase's open-source tool if you have SQL skills or multiple databases. For in-app mobile event tracking she recommends CleverTap over Mixpanel (which failed her) and Amplitude (which lacks the CRM components). Bigger needs point to Amplitude plus Segment, and for experimentation she shouts out Eppo to automate the decision-making flow.

  • Single warehouse / very early: Google Data Studio (free) or Metabase (open source, needs SQL).
  • In-app mobile event tracking: CleverTap preferred over Mixpanel and Amplitude.
  • At larger scale: Amplitude for analytics juice plus Segment to reduce ETL.
  • Experimentation: Eppo to automate the decision-making flow.

if you need in-app mobile device event tracking i usually recommend clevertap because mixpanel has unfortunately failed me a lot and amplitude doesn't have the…

Crystal Widjaja · 47:30
#tools#analytics#stack#experimentation

Takeaway· 2

Takeaway25:00

Concrete Retention Benchmarks: 60% Free, 20-30% Paid, 80% Friends

Crystal shares specific week-over-week retention targets. Free products should hold at least ~60% weekly; paid products more like 20-30%. At tiny scale, friends-and-family retention should be near 80% because if you can't convince people who care about you, the product likely doesn't solve the job. She thinks in cohorts: week-one retention should hit ~60% and then flatten within two to three periods.

  • Free product: at least ~60% week-over-week retention at scale.
  • Paid product: roughly 20-30% weekly retention.
  • Friends-and-family / tiny scale: near 80%, or the product isn't solving the job.
  • Thinks in cohorts: week-one ~60%, then flatten over 2-3 frequency periods; Gojek early days saw 60-70%.

if it's a free product 60 percent right it has to be at least 60 if it's a free product week over a week if…

Crystal Widjaja · 25:00

that better be near close to 80 no matter what because if you can't even convince the people who care about you to use the…

Crystal Widjaja · 25:30
#retention#benchmarks#metrics#cohorts
Takeaway48:30

Growth as the "Cleanup Crew" of Stats-Heavy People

Crystal describes how Gojek's growth team formed by filling obvious gaps the core product team left behind (phone-number masking, SMS/OTP delivery, first-time-user education) rather than inventing a mandate. The team called itself the cleanup crew, staffed heavily with statistics graduates who cared about numbers and odds. Her warning: the worst outcome is a growth person who thinks they're right but is measuring wrong and chasing the wrong areas.

  • Growth emerged by filling gaps the core product team couldn't cover, not from a grand plan.
  • Examples: phone-number masking, choosing SMS/OTP providers, educating first-time users.
  • The team was stats-heavy (statistics graduates) because scale makes it a numbers game.
  • Worst case is a growth hire who measures things wrong and focuses on the wrong areas.

we really called ourselves like we're the cleanup crew we pick up the pieces that were left behind we connect the dots

Crystal Widjaja · 52:00

the worst possible thing is to have a growth person who thinks they are doing the right thing and is measuring things wrong and then…

Crystal Widjaja · 53:00
#growth-team#hiring#org-design#statistics