Ship-to-Learn: The Emergent-Product Loop
When product properties are emergent, launching is how you discover them
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 93%
For AI products, the capabilities and best use cases are emergent — not knowable in advance by reasoning. So the sequence inverts: ship something deliberately open-ended, watch what people actually do, then systematically improve the use cases that emerge. Polish comes AFTER launch because you literally cannot know what to polish until you see real usage.
Origin
Turley describes this as 'a pattern with AI.' Every bespoke prototype OpenAI built (meeting bot, coding tool) got hijacked by users wanting to do other things, because the tech is generically powerful. So they shipped ChatGPT open-ended purely to capture real use-case distribution, expecting to wind it down after gathering data.
Core principles
- 01The model is the product — iterate on it like software, not like hardware R&D
- 02You won't know what to polish until after you ship
- 03Open-ended launch captures real use-case distribution you can't predict
- 04Shipping is one point on the journey to awesomeness, not the end — but you must follow through and polish once you know what people do
- 05Failure cases from real usage are the raw material for improving the model
How to run it
- 1
Ship something deliberately open-ended
Resist over-scoping to a bespoke use case. Launch a general surface so real users reveal the distribution of what they actually want.
Pro tip No waitlist, launched for everyone at once, lets you watch the whole market use it live and learn from each other.
Watch out Do not use 'it's emergent' as a permanent excuse to never polish — that is the failure mode.
- 2
Stop and watch the real usage
After launch, deliberately pause to observe utility AND risks. With emergent products, missing this step means missing most of what the product actually does.
Pro tip Turley protects one full day a week for thinking/processing precisely because this observation is easy to skip when moving fast.
- 3
Cluster the emergent use cases
Identify what people are actually trying to do — write, code, get advice, get recommendations — using data-science classifiers plus qualitative interviews.
Pro tip Stop interviewing when you can predict what the next person will say; until then keep going.
- 4
Systematically improve the top use cases
Treat discovered use cases as a product backlog. Feed real-world failure cases to the model/eval teams as concrete targets to climb.
Pro tip Real failure cases beat saturated benchmarks — 'people are trying to do X and the model's failing in ways Y, now make those good.'
- 5
Now polish — intentionally
Once you know what people do, there is no excuse not to polish response layouts, UI, and formats. Pick your ship point intentionally; make it the start of iteration, then follow through.
Watch out The model chooser dropdown is Turley's own admitted example of shipping raw — acceptable to learn, but you owe the follow-through.
In the wild
OpenAI shipped a funky, unpolished capability before it was refined. Only after real-world use came back could they optimize it into what is now the data-analysis feature.
→ Shipping raw generated the real use cases that made the refined version possible.
Turley openly owns that the giant model-selection dropdown is 'the anti-pattern of any good product,' but argues shipping it raw to start learning was strictly less bad than waiting for polish.
→ Learning captured now; cleanup follows — he expects to be roasted until it's fixed.
Common mistakes
Reasoning about use cases a priori
With emergent AI products you cannot deduce what people will do. Every bespoke OpenAI prototype got repurposed by users, proving the point.
Polishing before you ship
You'll polish the wrong things. The properties of the product aren't knowable in advance, so pre-launch craft is often wasted effort.
Using emergence as a permanent excuse to never polish
Turley warns it's easy to weaponize 'it's emergent' to avoid ever building a great product. Ship is the beginning of iteration, not a license to stay raw.
Is it for you?
Best for
PMs and founders building AI products where the capability surface and use cases are genuinely unknown until users touch it
Not ideal for
Domains with known specs and high cost of a bad first impression (e.g. safety-critical, regulated, or hardware)
From the transcript
“this is a pattern with AI I think where you know you really have to ship to understand what is even possible and what people…”
“the one thing we've learned with Chad CBT is that there really is no distinction between the model and the product. Like, the model is…”
“You absolutely should polish, you know, um things like the model output, etc., but you won't know what to polish until after you ship.”
“shipping is just kind of one point on the journey towards awesomeness and you should put pick that point uh intentionally”
From the episode
Inside ChatGPT: The fastest-growing product in history
Nick Turley (Head of ChatGPT at OpenAI)