Continuous Calibration, Continuous Development (CCCD)
A CI/CD-style loop for non-deterministic AI: scope, evaluate, deploy, then calibrate against surprises.
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 4
- Confidence
- 93%
A development lifecycle purpose-built for AI products, named as the AI analog to CI/CD. It pairs a 'continuous development' arm (scope capability, curate data, set evaluation metrics, deploy) with a 'continuous calibration' arm (analyze real behavior, spot error patterns, apply fixes, design new metrics). Use it as your operating model whenever you build on a non-deterministic LLM API.
Origin
Coined by Aishwarya Reganti and Kiriti Badam after building end-to-end agents that collapsed under emergent errors (a customer-support product they had to shut down) and after incidents like Air Canada's agent hallucinating a refund policy the company was legally forced to honor. The name deliberately echoes CI/CD.
Core principles
- 01Evaluation metrics only catch errors you already anticipated; production surfaces the emergent ones.
- 02Curating an expected-input/expected-output dataset up front exposes team misalignment on how the product should behave.
- 03Not every error deserves a new eval metric — spot, one-off errors (e.g. a poorly defined tool call) can just be fixed and forgotten.
- 04Run iterations at low agency / high control early, increasing agency as the flywheel matures.
- 05Advance stages by 'minimizing surprise', not by a fixed rulebook.
How to run it
- 1
Scope capability and curate data (Development)
Before building, assemble a dataset of expected inputs and the outputs the AI should produce. This alignment exercise reveals where PMs, engineers, and subject-matter experts disagree on intended behavior.
Pro tip It doesn't need to be comprehensive — it just needs to let you get started and to force behavioral alignment.
Watch out Skipping this means shipping without a shared definition of 'correct.'
- 2
Set up the app and design evaluation metrics
Stand up the application and define evaluation metrics — the specific dimensions (e.g. verbosity, refusing refunds it shouldn't) you will measure during the evaluation process.
Pro tip Say 'evaluation metrics' not 'evals' — evaluation is the process, metrics are the dimensions; the term 'eval' has been diluted into meaninglessness (Martin Fowler's 'semantic diffusion').
Watch out Do not assume you can predict up front whether you need an LLM judge versus implicit production signals.
- 3
Deploy and monitor production (Calibration)
Ship it and watch real behavior via explicit signals (thumbs up/down) and implicit ones (a user regenerating an answer = the first one missed). Use these to decide which traces to inspect when volume is too high to review all.
Pro tip For high-throughput products you can't review every trace — let implicit/explicit signals point you to the traces worth examining.
- 4
Spot error patterns and recalibrate
Examine failure patterns in the flagged traces. For patterns you care about, build a new evaluation dataset/metric, ship a fix, and check for regressions. For spot errors, just fix and move on. Recalibrate whenever a model swap or shifting user behavior throws calibration off.
Pro tip Advance to more autonomy only when calibration (every 1-2 days) stops revealing new data-distribution patterns.
Watch out Deprecations (e.g. GPT-4o to 5) and evolving user behavior silently break calibration — treat them as triggers to re-run the loop.
In the wild
A system helped underwriters pick policies from 30-40 page loan applications; for 3-4 months users reported strong time savings. Then, excited, they began throwing whole applications at it asking 'for a case like this, what did previous underwriters do?' — a natural extension for users but a fundamentally different build requiring historical-document retrieval and interpretation of 'a case like this.'
→ Demonstrated that user behavior evolves post-deployment, forcing a recalibration and rebuild the original dataset never anticipated.
Each CCCD iteration (routing, then draft-suggestion co-pilot, then end-to-end resolution) yields specific learnings fed back into the loop: better routing data, context-engineering structure, logged human edits as free error analysis.
→ A low-risk flywheel that improves the product without a big-bang autonomous launch.
Common mistakes
Relying on evals alone as the whole feedback loop
Evaluation metrics catch only known failure modes; emergent patterns only appear in production, so an eval-only strategy misses the errors that actually erode trust.
Over-building LLM judges for complex use cases
In complex domains, emerging patterns outrun any judge; you end up building too many evals. Sometimes it's better to watch user signals, fix, check for regression, and move on.
Is it for you?
Best for
AI product teams (PM + engineer + SME sitting together) operationalizing a non-deterministic product past the prototype stage.
Not ideal for
Fully deterministic software, or one-off internal scripts where there is no ongoing user-behavior distribution to calibrate against.
From the transcript
“we came up with this idea of continuous calibration continuous development. The idea is pretty simple which is um we have this right side of…”
“evaluation metrics catch only the errors that you're already aware already aware of. But there can be a lot of emerging patterns that you understand…”
“There's not really a rule book you can follow, but it's all about minimizing surprise”
From the episode
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon