Error Analysis: Open Coding to Axial Coding to Count
Turn messy LLM logs into a prioritized list of failures before you write a single test.
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 95%
A three-stage, data-first process for figuring out what to actually improve in an AI product. You read real production traces one at a time and jot free-form notes (open coding), use an LLM to cluster those notes into named failure modes (axial coding), then count how often each failure occurs to see what to fix first. It replaces guessing and vibe-checks with evidence, and grounds any tests you later build in real behavior rather than imagined requirements.
Origin
Hamel Husain and Shreya Shankar are explicit that they did not invent this. Open coding and axial coding are terms of art from social-science qualitative analysis (grounded theory); error analysis itself is a decades-old machine-learning technique, which Hamel illustrates with an 8-year-old Andrew Ng video. Their contribution is adapting these established methods specifically to LLM applications.
Core principles
- 01Start with data analysis to ground what you should even test — never jump straight to writing tests
- 02Product/domain people must be in the room because the conversation IS the product
- 03Capture only the first, most upstream error in each trace; don't try to catalog every problem
- 04Do not build your taxonomy while coding — get down what's wrong first, then organize
- 05Basic counting is the highest-leverage analytical technique; simple beats sophisticated to start
- 06An LLM cannot do the open-coding step for you because it lacks the product context to spot bad 'product smell'
How to run it
- 1
Load traces in any observability tool
Pull real production traces (the full logged sequence of system prompt, user turns, tool calls, RAG retrievals, and AI responses) into an observability tool. The specific tool doesn't matter — Braintrust, Phoenix/Arize, Langsmith all work.
Pro tip Build or vibe-code a lightweight custom annotation UI that hides the system prompt by default and gives you a hotkey to advance traces — removing friction makes the highest-ROI activity something you'll actually keep doing.
- 2
Open code: read one trace, write one note
Read a trace with your product hat on and write a short, informal note describing the first thing that's wrong. Put on domain judgment — a technically 'correct' answer (e.g. 'we don't have that, have a nice day' to a lead) can still be a product failure worth noting.
Pro tip Be specific enough that the note stands on its own later — 'did not confirm call transfer with user' is usable; 'janky' is not, because neither an LLM nor a human can categorize it later without re-reading the trace.
Watch out Capture only the most upstream error, then stop and move on. Chasing every error in a single trace makes the process infinitely expensive.
- 3
Keep going until theoretical saturation
Sample and label traces until you stop uncovering new types of failures — a qualitative-research concept called theoretical saturation. Use '100 traces' as a mental target to unblock yourself, but the real stopping rule is when new traces stop teaching you anything.
Pro tip The number is often far lower than 100 (people find 15-60 enough). After two or three rounds you develop the intuition for when you've saturated.
- 4
Axial code: cluster notes into failure modes with an LLM
Export your notes to a CSV and ask an LLM (Claude, ChatGPT, Julius, etc.) to group the 'open codes' into 'axial codes' — named failure-mode categories. Using the actual terms 'open codes' and 'axial codes' in the prompt shortcuts the LLM to the right behavior because those are established concepts.
Pro tip Add a 'none of the above' category so the LLM can flag notes that don't fit — that signals your categories are incomplete and need reworking.
Watch out The LLM's first-pass categories will be too generic to act on (e.g. 'capability limitations'). Rename them into specific, actionable failure modes yourself before proceeding.
- 5
Re-label every note against your refined categories
Once you've refined the category list, feed it back and have the LLM assign each original note to exactly one category (a simple spreadsheet-AI formula works). You must re-map because you changed the categories after the LLM first proposed them.
Pro tip Stay in the loop — review the refined codes and ask whether they actually make sense to you before mass-labeling.
- 6
Count with a pivot table and prioritize
Pivot-table the labeled data to count how often each failure mode occurs. Now you've gone from chaos to a ranked view of your biggest problems (e.g. '17 conversational flow issues'). Attack the most prevalent — but weight by severity too, since a rare failure like a bad human-handoff may matter more than a frequent minor one.
Pro tip This is largely a one-time cost. Once the categories exist, re-running the whole loop weekly takes about 30 minutes.
Watch out Don't let count be the only signal. Some low-frequency failures are business-critical and jump the queue despite low counts.
In the wild
Hamel walked through real anonymized traces from Nurture Boss, an AI assistant for apartment property managers. Reading traces surfaced failures no one had flagged: the AI told a lead 'we don't have a study, have a nice day' instead of handing off to a human; text-message conversations got 'janky' because users split sentences across turns; and the AI hallucinated a virtual tour that didn't exist. An LLM asked 'is there an error?' called these traces fine — it lacked the context to know the tour didn't exist.
→ A handful of distinct, previously-invisible failure modes were identified and counted in a short session, giving the team a concrete, evidence-based fix list. Nurture Boss later built their own annotation UI with automated axial-coding counts in a few hours.
Reviewing a client's recruiting-email agent, Hamel immediately reacted to an email opening with 'given your background...' — a phrase he personally deletes on sight as generic spam. The client was actually proud of the output because the AI had correctly assembled the right name, link, and information.
→ Applying a product-taste lens (not just technical correctness) exposed a quality problem the builder had been blind to, redirecting the fix toward output quality rather than mechanics.
Common mistakes
Jumping straight to writing tests
People skip data analysis and start writing eval tests based on imagined requirements. This is where things 'go off the rails' — you end up testing failures that aren't your real failures. Ground tests in error analysis first.
Trying to automate the open-coding step with an LLM
The number-one pitfall. An LLM lacks the product/domain context to judge whether something is a real failure, so it rubber-stamps bad traces as 'looks good.' Humans must do the trace-reading; LLMs only help with clustering and counting afterward.
Writing vague open codes
Notes like 'janky' can't be categorized later by an LLM or a human without re-reading the trace, breaking the whole automated axial-coding step.
Is it for you?
Best for
Product managers and engineers building an LLM/agent product who feel lost relying on vibe-checks and don't know what to fix first.
Not ideal for
Coding-agent teams where the developer is themselves the domain expert and dog-foods all day, so they can collapse much of the data-gathering into their own usage.
From the transcript
“You should start with some kind of data analysis to ground what you should even test.”
“the first step in conquering data like this is just to write notes.”
“the answer is just write down the first thing that you see that's wrong, the most upstream error.”
“the right answer is keep looking at traces until you feel like you're not learning anything new”
“basic counting is the most powerful analytical technique in data science”
“We didn't invent error analysis. We don't actually want to invent things.”
From the episode
Why AI evals are the hottest new skill for product builders
Hamel Husain & Shreya Shankar (creators of the #1 eval cours