Ship Fast Without Breaking Money
Reject the reliability/velocity trade-off: automated gauntlet, ramped exposure, and remediations ahead of roadmap.
- Difficulty
- Expert
- Time to result
- ~ongoing to results
- Steps
- 6
- Confidence
- 92%
Most business-critical systems buy reliability by changing as little as possible. Stripe holds five-nines uptime while deploying to its core API 16.4 times a day, by treating reliability as a systems problem rather than a caution problem. Every change runs an automated test gauntlet, auto-deploys within ~45 minutes, ramps from a small traffic percentage to the whole, and every incident produces remediations that are prioritised ahead of roadmap work.
Origin
David Singleton, CTO of Stripe, describing how Stripe holds 99.999% uptime and ~16.4 core API deploys/day simultaneously. Stripe deliberately built the approach from first principles rather than copying financial-services norms, while learning from companies like Amazon (Singleton met Charlie Bell, then running Amazon operations, in his first weeks).
Core principles
- 01Freezing change is a real way to be reliable, and it is the wrong one — user needs evolve too fast, and the tight user feedback loop only exists if you can ship.
- 02Reliability and velocity are held true simultaneously by systems, not by caution.
- 03No manual testers: no human could cover the array of API endpoints and configurations, but automated tests can.
- 04Design so that something breaking for one user does not carry over into other users.
- 05The learning obligation is not 'what would have stopped this incident' but 'how do we prevent this whole class of issue'.
- 06Remediation work outranks roadmap work, because unreliability is what ultimately stops you moving quickly for users.
How to run it
- 1
Build the automated test gauntlet
Every change an engineer produces runs through a battery of automated tests before it can go anywhere near production. Stripe employs no manual testers at all.
Pro tip Copilot is unusually good at generating the boilerplate in test cases, freeing engineers to reason about whether the test actually tests the right thing.
- 2
Run tests in parallel with human review, and again on merge
The suite runs once (~15 min) alongside the human code review, then the same suite runs again after merge (~15 min), then the systems auto-deploy (~30 min). Total: roughly 45 minutes from merge to production.
Pro tip Invest in selective test execution — figuring out which tests are material to a given change — and in parallelising the pre-production run across more machines. Stripe's distributed change-and-test environment is the largest distributed system it runs.
Watch out As you add product you add tests; these numbers only hold with continuous investment in the test infrastructure itself.
- 3
Auto-deploy — remove the human babysitter
Once tests pass, the change reaches production automatically. Before this, every deploy required an engineer to watch it go live and check the charts, and engineers hoped their change would get bundled with someone else's so that person would babysit it.
Pro tip Add an auto-merge checkbox on the pull request so that once the reviewer approves, the system takes over without another human step. This small change laddered up to a big productivity gain.
Watch out Auto-deploy requires a mind shift: you must assume it will happen and put the surrounding systems and processes in place first.
- 4
Ramp exposure through increasingly realistic environments
Push changes through staging environments running realistic end-to-end tests, then in production start at a very small percentage of traffic and ramp to the whole — so problems are detected before they become huge problems.
- 5
Contain blast radius by design
Assume things will break — downstream partners fail, networks fail outside your control. Build redundant systems and design so that a failure affecting one user does not carry over into others.
Pro tip Add chaos testing: inject errors deliberately and verify the systems respond without user impact.
- 6
Prioritise incident remediations ahead of the roadmap
Review every incident carefully, ask how to prevent the whole class of issue, and schedule the resulting remediations before other roadmap work. PMs therefore plan with reserved bandwidth for this.
Pro tip Ask each team to decide how much bandwidth to reserve for remediations and polish, rather than mandating a company-wide percentage — it varies by team and product stage.
Watch out An incident review that only asks 'what would have stopped this specific failure' guarantees you meet the same class of failure again in a different costume.
In the wild
Stripe deploys changes to its core API 16.4 times a day on average while sustaining five-nines uptime, moving as much money annually as all of e-commerce did when Stripe started. It does this with no manual testers, a ~45-minute merge-to-production pipeline, and traffic-percentage ramping.
→ The tight loop means a user can give feedback in the morning, the team can address it, and something can be back in their hands by the end of the day — a full learning cycle inside 24 hours.
Within the last five years, every Stripe deploy to production required an engineer to babysit it, watching the charts as it went live. Engineers played a kind of roulette hoping their change would be bundled with someone else's so that person would watch it for them. Stripe replaced this with automated deploys plus automated monitoring.
→ Engineers can keep their fingers on the keyboard on something else while their change goes live — the single most significant individual improvement to developer productivity Singleton names.
Common mistakes
Buying reliability with change-freezes
Changing infrequently does reduce breakage opportunities, but it also severs the user feedback loop and leaves rapidly-evolving user needs unserved. The cost is invisible on the uptime dashboard and enormous in the market.
Relying on manual testing at API scale
Manual testers cannot possibly cover the vast array of API endpoints and configurations. Any manual gate becomes either a bottleneck or a rubber stamp.
Treating remediations as backlog items
If remediation work queues behind feature work, the same class of incident recurs, and the resulting unreliability is precisely what eventually stops you shipping quickly for users.
Is it for you?
Best for
Engineering leaders at companies where the product is business-critical to customers (payments, infrastructure, health, financial data) and where the instinctive answer to risk is to slow down.
Not ideal for
Early-stage teams with no test suite and no traffic — the infrastructure cost of this pipeline vastly outweighs the benefit before you have real load and real stakes.
From the transcript
“there's one way to be very reliable, which is to try to change things as infrequently as possible”
“Then finally once something actually goes to production, it starts at a very small percentage of the traffic and then ramps up to the whole.”
“we identify instant remediations. That's what we call those. And we prioritize those carefully, and most of them, the ones that matter, actually get prioritized…”
“reviewing them carefully and identifying not only like what would have stopped this thing happening, but how can we prevent this whole class of issues…”
From the episode
Building a culture of excellence
David Singleton (CTO of Stripe)