Protected Deep-Dive Time
Goal the team on self-directed insight and use hackathons to stop exploratory work being eaten by inbound asks
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 93%
Exploratory work is always the first thing to go: a known request with a certain payoff beats a deep dive with unknown ROI, every single day. Lachs's answer is to make self-directed insight an explicit goal (an accountability mechanism, not an aspiration) and to carve out whole days via hackathons. In the deep dive itself, the discipline is to distrust averages, decompose distributions, and go as far as personally abusing your own product to find out what is really happening.
Origin
Jessica Lachs, DoorDash analytics — most vividly demonstrated by a referral-channel hackathon in which the team committed referral fraud against themselves and ordered cupcakes to the office.
Core principles
- 01Exploratory time never survives without a mechanism — it must be goaled, not merely encouraged
- 02The expected value of a known ask always looks higher than an unknown deep dive, so intent must override instinct
- 03The average is often incredibly misleading — look at the distribution
- 04Don't lower spend on an underperforming channel until you understand why it underperforms
- 05Business partners will protect this time once deep dives have visibly driven roadmaps
How to run it
- 1
Set goals around self-directed insight
Add finding insights through self-directed work to the team's formal goals. This is the accountability mechanism that stops exploratory time from silently evaporating under inbound pressure.
Watch out Without a goal, the deep dive loses every prioritisation call to the one-hour data pull that makes someone happy today.
- 2
Carve out whole days via hackathons
Run hackathons that block out full days for the team to chase interesting threads. Partial hours get reclaimed by inbounds; days do not.
Pro tip Rotate the themes so different parts of the business get the treatment.
- 3
Pick a metric that looks bad and refuse to act on the average
Choose something underperforming — a below-average acquisition channel, a soft payback period — and resist the obvious action (cut spend) until you understand the mechanism behind the number.
Pro tip The instinct to cut and move on is exactly what buries the insight.
- 4
Decompose the distribution
Break the average into its distribution. Bimodality frequently hides two opposite populations whose blended mean tells a story that is true of neither.
Pro tip If the distribution is bimodal, the right action is usually 'do more of one, kill the other' — not 'do less of it'.
- 5
Use the product adversarially to see the mechanism
Go beyond the dataset: use the product, abuse it, try to break the rules yourself. Create accounts, try to game your own incentives, and see what the data can't tell you.
Pro tip Convert the findings into concrete recommendations (rules, caps, checks), not just an interesting deck.
In the wild
Referral looked below-average on engagement and payback, and the obvious move was to cut spend. Instead, in a hackathon, the team referred each other, deliberately committed referral fraud, created new accounts to get around the rules, and ordered so many cupcakes to the office (an order was required to trigger a referral bonus) that they learned exactly how the channel was being gamed. The distribution turned out to be bimodal: excellent consumers referring excellent consumers, and a second population arriving via referral codes posted online purely for free credit.
→ DoorDash tightened fraud checks and added caps on referrals rather than cutting the channel — preserving the high-payback half of a channel it would otherwise have suppressed, and proving to business partners that deep-dive time drives roadmaps.
Because so many insights that shaped future roadmaps came out of these deep dives, DoorDash's business partners now actively encourage the analytics team to take self-directed exploration time.
→ Exploratory time became politically protected rather than something the data team had to defend each quarter.
Common mistakes
Treating exploration as a cultural aspiration
Encouragement loses to inbound requests every time, because a known ask has a certain payoff and a deep dive has an unknown one. Only a goal, and blocked-out days, survive contact with the queue.
Acting on a below-average metric without understanding it
Lowering spend on an underperforming channel and moving on can destroy the profitable half of a bimodal population while leaving the abusive half untouched.
Stopping at the dashboard
The referral fraud was only visible because the team used and abused the product themselves. Some mechanisms cannot be seen from inside the dataset.
Is it for you?
Best for
Data and analytics leaders whose teams are drowning in inbound requests and never get to the proactive, opportunity-finding work they were hired for
Not ideal for
Teams that have not yet earned credibility on the basics — if core reporting is broken, exploratory hackathons will read as avoidance
From the transcript
“you have to be very intentional to carve out time for exploratory work for deep Dives”
“setting goals for your team around finding this these insights through self-directed work is an important mechanism for holding ourselves accountable to that goal”
“we would do hackathons for our team to carve out days to just go and look into these really interesting things”
“we actually tried referring each other we tried committing referral fraud creating new accounts to get around rules”
“the average can be incredibly misleading and so looking at distributions”
From the episode
Building a world-class data org
Jessica Lachs (VP of Analytics and Data Science at DoorDash)