Vibes Before Evals
For a genuinely new AI feature, start with open-ended vibes testing; add evals only once the use-case cluster is clear.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 90%
A sequencing rule for building novel AI products. Rigorous evals are powerful, but for a completely new product experience or form factor you should NOT start with them — you should start with vibes: open-ended, ad-hoc testing where you throw varied prompts at the capability to discover, broadly, whether it even works and which use cases it's good at. Evals become useful only after you've converged on the basic form-factor scaffold and know the cluster of use cases worth testing against; then you can measure improvements programmatically and A/B test.
Origin
Howie Liu's additive take, offered in explicit dialogue with the eval-first advice from the heads of OpenAI and Anthropic that Lenny had featured. Lenny maps it to the 'double diamond' diverge-then-converge design framework (IDEO), which Liu affirms.
Core principles
- 01For a novel form factor, evals defined too early constrain you before you know what 'good' even looks like.
- 02Vibes testing is a divergent discovery process; evals are a convergent iteration-to-improvement tool.
- 03You must first use your own brain to imagine the range of use cases, then see empirically where the capability does well.
- 04At sufficient scale (OpenAI/Anthropic) you can A/B test everything — but early on you lack that luxury and must explore.
How to run it
- 1
Diverge with ad-hoc vibes testing
For a genuinely new capability, throw varied prompts at it in an open-ended way. Use your own judgment to brainstorm the full range of use cases you can imagine, then run them and observe broadly whether and where it works.
Pro tip Deliberately test the edges and random cases, not just the happy path — you're mapping the capability's true footprint.
Watch out Defining repeatable evals at this stage locks you into a definition of 'good' before you understand the solution space.
- 2
Find the cluster of use cases it's good at
From the vibes results, notice where the capability clearly excels — e.g. a web-crawler agent that compiles lists does well on 'people and companies with this kind of parameter.' That cluster becomes the scope of the product.
Pro tip Let the observed strengths, not your original ambition, define how you'll merchandise the feature (e.g. a picker of supported entity types with explicit filter criteria rather than a fully open-ended box).
- 3
Converge to evals once the scaffold is stable
After you've settled the basic form factor and know the target use cases, define what 'good' looks like and build repeatable evals. Now vary the prompt, model, or agentic workflow and measure the effect on output quality programmatically.
Pro tip At large scale you can A/B test model and workflow variants empirically ('this model performs better than that one').
Watch out Skipping straight to this stage on a novel product means you're optimizing metrics for a form factor you haven't validated.
In the wild
For a new long-running AI crawler that compiles lists of entities (companies, people, 'every Marvel movie ever made'), Liu's team first tried a bunch of random use cases by hand to see what worked, discovering it did really well on people and companies with certain parameters.
→ Only after that open-ended discovery did evals become useful — the team scoped the product to a specific set of entity types with explicit filter criteria, then could programmatically measure improvements to output quality.
Common mistakes
Starting a novel capability with evals
Evals require a definition of 'good,' which you don't have for a brand-new form factor; committing to them early constrains exploration and optimizes the wrong thing.
Never graduating from vibes to evals
Staying purely ad-hoc once the use cases are clear leaves you unable to measure whether prompt/model/workflow changes actually improve output, so quality plateaus on gut feel.
Is it for you?
Best for
Product and AI engineering teams building a genuinely new AI feature or form factor without an existing 'good' benchmark.
Not ideal for
Incremental improvements to an existing, well-understood capability, where the use cases and success criteria are already known and evals should lead.
From the transcript
“for a completely novel product experience or form factor, you should actually not start with Evals and you should start with Vibes”
“eval are more useful a once you've converged on the kind of like basic scaffold of the form factor and you kind of know what…”
“you have to just be a little bit more creative initially and like throwing stuff at it, seeing what works”
From the episode
How we restructured Airtable’s entire org for AI
Howie Liu (co-founder and CEO)