Decompose-and-Ensemble
Break a broad problem into specific tasks, then solve each with a specialized model in an ensemble.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 4
- Confidence
- 90%
Instead of handing a model one broad, generic problem, break the problem down into specific sub-tasks and solve each with the right model — specialized fine-tuned models, different sizes for different latency/cost needs, and custom prompts per task — then ensemble the results. Weil likens it to how a company is an ensemble of people, each fine-tuned by their training, producing a better output together than any individual could alone.
Origin
Kevin Weil (CPO, OpenAI) describing how OpenAI operates internally, contrasted with how many companies still give models single generic broad problems.
Core principles
- 01Breaking a problem into specific tasks beats asking one model a broad high-level question
- 02Use different model sizes for different latency and cost requirements
- 03Use custom prompts and fine-tuned models for each specific sub-task
- 04The ensemble's output is better than any single model's — like a company of specialized people
How to run it
- 1
Break the problem into specific tasks
Decompose the broad goal into narrower, well-defined sub-tasks rather than one high-level ask.
Watch out Giving a model a single generic broad problem is the common mistake that leaves performance on the table.
- 2
Assign the right model to each task
Match each sub-task to a model: a reasoning model where quality matters and latency is tolerable, a fast cheap model (e.g. a mini) for quick checks, and fine-tuned models where you have use-case data.
Pro tip Solving 10 problems may take 20+ model calls of varying sizes — that's expected, not wasteful.
- 3
Write custom prompts per task
Tailor a prompt (and examples) to each sub-task so each model is very good at its specific job.
- 4
Ensemble the outputs
Combine the specialized outputs — often with a model integrating them — into a single result that beats any one model's answer.
In the wild
With 400M+ weekly active users but only ~30-40 support staff, OpenAI automates most tickets: the model answers common questions from internal knowledge bases and guidelines; where it lacks confidence it drafts an answer and requests a human, whose correction becomes fine-tuning data. It uses reasoning (o-series) models where latency is tolerable and fast cheap models (4o-mini) for quick checks.
→ A support operation far smaller than a comparable company's, because flows are automated with specific models for specific purposes.
Common mistakes
Giving the model one broad generic problem
Many companies ask a single baseline model a high-level question instead of decomposing it and using specialized models per task, leaving performance unrealized.
Is it for you?
Best for
Engineering and ML teams building production AI systems that need reliability, low cost, or low latency
Not ideal for
Simple one-shot tasks where a single general model already performs well and decomposition adds needless complexity
From the transcript
“break the problem down into more specific tasks versus some broader set of highle tasks”
“specific models for specific purposes and then you you you ensemble them together to solve problems”
“A company is arguably an ensemble of models that have all been,”
From the episode
OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more
Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter)