Ensembling (Mixture of Reasoning Experts)
Solve the same problem several ways and take the most common answer
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
An advanced technique: send the same problem through multiple prompts (different techniques, roles, or even different models), collect the varied answers, and take the most frequently returned answer as final. It mirrors polling several experts and taking the consensus. A named variant is Mixture of Reasoning Experts, where each 'expert' is an LLM prompted differently or given different tools/internet access.
Origin
Rooted in classical ML ensembling (e.g. random forests). Mixture of Reasoning Experts was developed by a colleague of Schulhoff's now at Stanford. Catalogued in The Prompt Report.
Core principles
- 01Multiple independent attempts plus majority vote beats a single attempt on accuracy tasks
- 02Different prompts/roles can activate different regions of the model and change performance
- 03Best measured on tasks with programmatically checkable answers (math, reasoning)
- 04Experts can be the same model prompted differently, different models, or models with different tool access
How to run it
- 1
Assemble a set of solvers
Choose several prompts or 'experts' — e.g. one with chain-of-thought, one told to act as a domain expert, one with internet access.
- 2
Run the same problem through each
Send the identical problem to every solver independently and collect all the returned answers.
Watch out Roles may or may not change accuracy — treat them as one lever among several, not a guaranteed boost.
- 3
Take the majority answer
Select the answer that comes back most commonly across the solvers as your final output.
In the wild
The same factual question is routed to an 'English professor' expert, a 'soccer historian' expert, and a no-role expert with internet access. The soccer historian and internet-search expert both return 13; the English professor returns 4. The majority answer, 13, is taken as final.
→ Majority voting across differently-prompted experts yields the more reliable answer.
Common mistakes
Relying on a single prompt for accuracy-critical tasks
One attempt can land on an outlier answer; ensembling and voting smooths this out.
Is it for you?
Best for
Product teams optimizing accuracy on programmatically-verifiable tasks where extra compute is justified
Not ideal for
Latency- or cost-sensitive flows, or subjective/generative tasks with no checkable 'correct' answer
From the transcript
“you'll have like multiple different prompts that go and solve the exact same problem”
“I kind of take the most common answer as my final answer”
“a specific example uh of one of these techniques is called mixture of reasoning experts”
“they can kind of activate different regions uh of the (44:00) model's neural brain”
From the episode
AI prompt engineering in 2025: What works and what doesn’t
Sander Schulhoff (Learn Prompting, HackAPrompt)