Evals-Plus-Production-Monitoring Dual Feedback Loop
Reject the false dichotomy: evals catch what you know, production monitoring catches what you don't.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
A decision framework for building an AI quality feedback loop that refuses the 'evals OR vibes/monitoring' debate. Evals encode your product knowledge about failures you can foresee; production monitoring surfaces the failures you cannot. Use both, in a loop, where monitoring signals tell you which failures deserve a new eval. Apply it when deciding how to guarantee reliability for a deployed AI product.
Origin
Kiriti Badam (OpenAI Codex) framed this against the public debate sparked when Claude Code's Boris said they run on 'all vibes,' contrasted with those who treat evals as everything. Kiti describes how Codex actually balances evals, customer-behavior watching, and vibes.
Core principles
- 01It's a false dichotomy that either evals or production monitoring alone will solve everything.
- 02Evals are your trusted product knowledge encoded as datasets of what must not go wrong.
- 03Production monitoring reads both explicit signals (thumbs up/down) and implicit ones (regenerating an answer, switching off a feature).
- 04Monitoring points you to the traces worth examining when you can't review them all.
- 05Even after building an eval for a caught failure, you still need monitoring to catch the next unknown one.
How to run it
- 1
Build a starter eval set before deploying
Nobody deploys blind. Encode the handful of things that must never go wrong (e.g. 10 questions it must not fail regardless of changes) into an evaluation dataset — this may start as vibes.
Pro tip Protect what is core to the product first, so a model or prompt change can't silently damage it.
Watch out Don't claim you 'do evals' when you only checked LM Arena or Artificial Analysis — those are benchmarks for model selection, not evals of your product.
- 2
Deploy and monitor implicit + explicit signals
Instrument the deployed product to capture explicit feedback and, crucially, implicit signals like answer-regeneration or users switching a feature off when it annoys them.
Pro tip For a code-review product, watch whether users turn it off after incorrect flags — a strong negative signal even without a thumbs-down.
Watch out With AI agents you must monitor at far higher granularity than traditional products; explicit feedback alone is not enough.
- 3
Turn recurring failures into new evals, then re-monitor
Examine failure patterns from monitoring; for ones you truly care about (e.g. an agent offering refunds when configured not to), build an eval dataset, ship a fix, verify no regression — then keep monitoring for the next unknown failure.
Pro tip A/B test model changes and gather custom hard-problem evals per engineer for each new model launch.
Watch out There's no guarantee your new eval covers the only problem; monitoring must stay on permanently.
In the wild
Codex ships evals to protect core behavior but leans heavily on watching real usage — A/B testing model changes, checking whether reviews catch real bugs, and reading social media to spot problems fast — because a customizable coding agent has too many integration paths to fully pre-enumerate in evals.
→ Caught many bugs internally and externally; balances 'a lot of vibes and a lot of customer feedback' rather than betting on evals alone.
For every new model, the team gathers, each person tests different hard problems from a shared list, and they observe how the model progresses — custom evals per engineer.
→ No single person claims a concrete eval set they can 'bet their life on'; coverage comes from distributed testing plus monitoring.
Common mistakes
Believing one approach solves everything
Betting entirely on evals misses emergent production failures; betting entirely on monitoring means you ship known-bad behavior — both extremes are dismissible.
Calling benchmarks 'evals'
Checking LM Arena / Artificial Analysis to pick a model is model selection, not evaluating your own product's behavior — 'semantic diffusion' has blurred the term.
Is it for you?
Best for
AI engineers and PMs setting up a reliability/quality process for a deployed, high-traffic AI product.
Not ideal for
Very early prototypes with no real users yet, where there is no production traffic to monitor.
From the transcript
“I feel there's this false dichotomy of like there's either eval is going to solve everything or online monitoring or production monitoring is going to…”
“sometimes customers don't give you thumbs down but actually re regenerate the answer. So that is an clear indication that the initial answer that you…”
“I feel eval are important, production monitoring is important but this notion of only one of them is going to solve things for you that…”
From the episode
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon