“They teach the definitive online course on evals, which happens to be the number one course on Maven.”
← All resourcescourse
AI Evals for Engineers and Product Managers
By Hamel Husain and Shreya Shankar
1 recommend/use · 1 sourced episode
Every sourced reference
Short attributed excerpts only. Timestamps are approximate.
“If this conversation gets you excited, definitely check out Hamill and Shreas's course on Maven.”
“If you Google AI evals for engineers and product managers or just AI evals course, you'll find it.”
Related frameworks
Innovation6 steps
Error Analysis: Open Coding to Axial Coding to Count
Turn messy LLM logs into a prioritized list of failures before you write a single test.
Read →
Strategy3 steps
Eval Triage: Decide What Actually Deserves an Eval
After counting failures, route each one to a prompt fix, a code check, or an LLM judge — not all three.
Read →
Innovation4 steps
The Aligned Binary LLM-as-Judge
Build a one-failure, pass/fail judge and align it to a human with a confusion matrix before you trust it.
Read →
Leadership3 steps
The Benevolent Dictator Labeling Model
Appoint one trusted domain expert to own eval judgments instead of running it by committee.
Read →
People in these episodes
Spot an error or want this page removed? Request a correction or removal.