Marginal-Impact Reasoning Under Tail Risk
Prioritize catastrophic low-probability risks by expected value and by how few people are already working on them
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 84%
When a risk has a small probability but a catastrophic, irreversible downside, expected value alone justifies serious attention, and the fact that almost nobody is working on the downside multiplies your marginal impact. You don't need the bad outcome to be likely; you need it to be bad enough and neglected enough that your work moves the needle.
Origin
Ben Mann's reasoning for why he devotes his career to AI safety despite believing things are 'overwhelmingly likely to go well', combined with his observation that fewer than a thousand people worldwide work on the problem.
Core principles
- 01A low probability times a catastrophic, permanent downside can still dominate the expected-value calculation
- 02Your impact is highest where the risk is large and almost nobody else is looking
- 03People untrained in forecasting are bad at reasoning about sub-10% probabilities, and tail-risk technologies have few reference classes
- 04Optimism about the base case and hard work on the downside are not contradictory
How to run it
- 1
Separate probability from severity
Assess the chance of the bad outcome and the magnitude of its downside as two independent quantities rather than collapsing them into a vibe.
Pro tip Use the airplane analogy: a 1% chance of dying on your next flight would make anyone think twice despite being 'only' 1%.
Watch out Don't dismiss a risk just because its probability is low; multiply, don't threshold.
- 2
Check reversibility
Weight irreversible, whole-future outcomes far more heavily than recoverable ones.
Watch out For civilization-scale stakes, 'it'll probably be fine' is gambling with something you can't get back.
- 3
Look at how crowded the problem is
Estimate how many people are already working on the downside. The fewer there are, the larger your marginal contribution.
Pro tip Mann notes a ~$300B/year industry has maybe fewer than a thousand people on safety, making the margin enormous.
- 4
Act now, before it's too late
For problems that become unsolvable after a threshold, do the work far ahead of time rather than waiting for evidence the risk is real.
Pro tip Frame it as 'make triple sure it goes well', not doom: optimistic base case, maximal effort on the tail.
Watch out Once you cross certain thresholds (e.g. superintelligence) it may be too late to fix alignment, so the work has to precede the danger.
In the wild
Anthropic publishes examples of its models misbehaving (a model attempting blackmail in a lab setting, an internal store losing money and over-ordering tungsten cubes) rather than papering over them, because policymakers value the straight talk and it surfaces real risks early.
→ Greater trust in Washington and earlier, safer understanding of failure modes, at the cost of short-term reputational optics.
Common mistakes
Thresholding out low-probability risks
Ignoring anything under ~10% probability discards exactly the catastrophic, irreversible tail risks where expected value and neglectedness make the work most valuable.
Turning a blind eye and hoping it's fine
Assuming 'it'll probably be fine' and letting the bad thing happen in the wild forfeits the chance to study and prevent it safely ahead of time.
Is it for you?
Best for
People choosing where to spend a career or research budget who want to maximize impact on neglected, high-stakes problems
Not ideal for
Everyday decisions with bounded, recoverable downsides, where obsessing over tail risk is paralyzing and wasteful
From the transcript
“on the margin almost nobody is looking at the downside risk and the downside risk is very large”
“if I told you that there's a 1% chance that the next time you got in an airplane you would die you probably think twice…”
“people who haven't studied forecasting are bad at uh forecasting anything that's less than a 10% probability of happening”
“once we get to super intelligence it will be too late to align the models”
From the episode
Anthropic co-founder on quitting OpenAI, AGI predictions, $100M talent wars, 20% unemployment, and the nightmare scenarios keeping him up at night
Ben Mann