RAG Data Preparation Over Database Tuning
The biggest RAG quality wins come from preparing data for retrieval, not picking a database.
- Difficulty
- Advanced
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 90%
A concrete toolkit for improving retrieval-augmented generation quality by restructuring the underlying data so it can actually be retrieved, rather than obsessing over vector-database choice. Use it when RAG answers are weak despite a solid stack. Covers chunk sizing, contextual metadata, hypothetical questions, Q&A reframing, and AI-specific annotation layers.
Origin
Explaining RAG (retrieval augmented generation), Huyen states that across the companies she's seen, the biggest performance gains in RAG come from better data preparation, not from agonizing over which database to use, then enumerates the specific preparation techniques.
Core principles
- 01Data must be prepared so the right chunk is retrievable even when the query terms don't literally appear.
- 02Chunk size is a deliberate trade-off between retrieval breadth and per-chunk relevance.
- 03Documentation written for humans often needs a separate annotation layer for AI, which lacks human common sense.
How to run it
- 1
Design chunk size deliberately
Tune how big each chunk is: too-long chunks limit how many you can retrieve; too-short chunks retrieve broadly but each may be too small to hold the relevant information.
Pro tip Treat chunk size as a tunable parameter balancing breadth vs. sufficiency, not a default.
- 2
Add contextual metadata
Attach summaries and metadata to chunks so a chunk that never repeats a key entity (e.g., a later section that drops the subject's name) is still retrievable for queries about it.
Pro tip The classic failure: a long doc chunks so that the entity name only appears once, making later chunks unretrievable.
- 3
Generate hypothetical questions
For each chunk, generate the questions that chunk could answer, so an incoming query can be matched to those hypothetical questions and fetch the right chunk.
- 4
Reframe into Q&A format
Rewrite source content (e.g., a podcast transcript) into question-and-answer pairs rather than raw chunks; you can use AI to do this at scale.
Pro tip One team cited this reframing as a very big performance win.
- 5
Add an AI annotation layer
Annotate documentation with context AI lacks: e.g., a doc saying a value should be 'one' means little to a model unless you explain the scale that value sits on.
Pro tip Human docs assume common sense; make the implicit explicit for the model.
Watch out AI reading differs from human reading; humans supply missing context that AI does not have.
In the wild
Instead of chunking a podcast transcript directly, a team rewrote it into many question-answer pairs (using AI), which produced a large retrieval-quality improvement.
→ Reported as one of their biggest RAG performance gains.
A library's docs said an output parameter should be a value like one/zero/minus-one; a human expert understood the scale but the AI didn't, so the team added an annotation layer explaining what the value means.
→ AI could retrieve and use the documentation correctly.
Common mistakes
Database-first thinking
Optimizing vector-database choice for answer quality when data preparation is the real lever; databases matter for latency and access patterns, not pure answer quality.
Chunking blind
Using a default chunk size without weighing the breadth-vs-relevance trade-off leaves relevant context unretrievable.
Is it for you?
Best for
Engineers whose RAG systems return weak answers despite a reasonable retrieval stack.
Not ideal for
Cases where latency or read/write access patterns, not answer quality, are the actual constraint.
From the transcript
“that's like the biggest performance in their rack solutions coming from like better data preparations not agonizing over like what better databases to use”
“a very big performance they got is that from um rewriting their data in the question answering format”
From the episode
Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)