Golden Samples Over Full Corpus
Curate a small set of gold examples for your AI, don't dump your whole knowledge base on it
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 2
- Confidence
- 93%
A data-preparation principle for building useful internal agents: connectivity and access are not the point — deliberate curation is. Pointing an agent at your entire drive makes it hallucinate and mis-weight the past; hand-selecting golden examples of good vs bad is what produces quality.
Origin
Framed by Tomer Cohen (LinkedIn CPO), who traces it back to over a decade earlier when rebuilding the LinkedIn feed and hand-filtering examples of good vs bad professional posts.
Core principles
- 01Access is not curation — most people wrongly focus on connectivity and integration
- 02An unfiltered corpus mis-weights importance and can't tie specs to outcomes
- 03Feeding in the right data beats feeding in all the data
- 04Building the golden sample is real work measured in weeks, not a one-click connect
How to run it
- 1
Decide the specific context and knowledge base per agent
For each agent, define exactly what context it needs and which knowledge base to focus it on, rather than granting blanket access to all company data.
Watch out Giving an agent access to your whole drive and telling it to reason over everything 'fails miserably and hallucinates like crazy.'
- 2
Hand-curate golden examples of good vs bad
Manually filter and label a gold-standard set that distinguishes good from bad for the domain, and tie examples to what actually succeeded, so the agent learns importance and outcome linkage rather than treating all data as equal fact.
Pro tip Budget real time — the LinkedIn feed golden sample took weeks of sit-down filtering, and it was the most important part.
Watch out An unfiltered corpus treats a strong-but-wrong researcher's opinion as fact and can't connect original specs to eventual success.
In the wild
Over a decade ago while co-rebuilding the LinkedIn feed from scratch, Cohen sat down and filtered through examples of what is and isn't a good professional post — weeks of work assembling the golden sample before training.
→ Fitting in the right data (not all the data) was the first and most important part of making the feed work.
Common mistakes
Connecting the full drive and letting the agent reason over everything
Unfiltered access makes the agent hallucinate, mis-weight the importance of past information, and treat contested opinions as fact — it fails miserably.
Focusing only on connectivity and integration
Many teams treat feeding an agent as a plumbing problem, but what you feed it — curated golden examples tied to outcomes — matters far more than mere access.
Is it for you?
Best for
Teams building internal domain agents on top of large, messy legacy knowledge bases
Not ideal for
Simple use cases with small, already-clean datasets where curation overhead isn't justified
From the transcript
“it's not great to just give it access to your drive and say reason all over this knowledge base. It actually does a very poor…”
“Just reasoning over your entire knowledge base does not work.”
“the first and most important part was fitting in the right data, not all the data”
“We talked about not just giving access to all of our context, which we started with. And like we were like, "Oh, here's access to…”
From the episode
Why LinkedIn is turning PMs into AI-powered "full stack builders”
Tomer Cohen (LinkedIn CPO)