Differentiate Above The Model, Not Around Its Gaps
Build your moat outside the LLM so your product gets better as the models get better.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 3
- Confidence
- 87%
When building AI products on non-deterministic LLMs, teams waste effort compensating for current model weaknesses. Yehoshua's rule: that compensating work is disposable because the models improve rapidly. Your durable differentiator must live outside the LLM, so that as models get smarter your whole product gets better rather than obsolete—and you add guardrails to teach users what the system can and can't do.
Origin
Tamar Yehoshua's product-strategy lesson from building Glean (enterprise AI search/assistant on top of LLMs).
Core principles
- 01The industry is transforming so fast that your product must get better as the LLMs get better.
- 02Work built to make up for or compensate for today's LLM weaknesses will go away—so it can't be your differentiator.
- 03As LLMs get smarter, everyone else's product becomes good too, so your edge must sit outside the model.
- 04Non-deterministic systems collide with enterprise expectations of deterministic software—users need educating.
- 05Chat interfaces are unfamiliar; guardrails are how you help users learn what will and won't work.
How to run it
- 1
Separate compensating work from durable value
Explicitly label the parts of your product that only exist to patch current LLM weaknesses. It's fine to build them—just know they're temporary.
Watch out If your differentiator is compensating for the LLM, it evaporates the moment the models improve.
- 2
Locate your moat outside the model
Identify the differentiator that continues to exist and improve as LLMs get smarter—for Glean, the enterprise knowledge graph indexing all your SaaS content. Build so the whole product rises with the models.
Pro tip Ask: 'as models get better and everyone else becomes awesome, what still makes us better than what other people are doing?'
- 3
Add guardrails to teach the user
Because chat interfaces are unfamiliar and users try queries the system can't possibly answer, build guardrails—autocomplete, refinements, prompt suggestions—to steer them toward what works, just as search took years and features to teach.
Pro tip Surface concrete prompt suggestions (e.g. 'here's a prompt to find the status of your feature') rather than leaving users to guess.
Watch out Enterprise users will experiment with ChatGPT on weekends but expect deterministic software at work—set expectations for non-determinism explicitly.
In the wild
Glean began as enterprise search (2019) using BERT models and vector embeddings from ex-Google engineers, indexing content across all a company's SaaS apps. When GPT-3 arrived it added a natural-language chat interface over what is effectively a knowledge graph of the organization. The LLM interface is replaceable; the enterprise-wide index and graph are the differentiator that improves as models improve.
→ Glean became one of the few AI companies actually making money, with the model layer as a swappable component atop a durable data moat.
In Yehoshua's first week at Glean, the assistant quality team's biggest issue was users asking things the system couldn't possibly know—like 'what should my top priority be next week' when Glean doesn't know your priorities. Search took years and features (autocomplete, refinements) to teach users to query well; chat needs the same guardrails.
→ The team's roadmap centers on guardrails and suggestions to bridge users to what the product can actually do.
Is it for you?
Best for
Product leaders building applications on top of rapidly-improving foundation models, especially in enterprise
Not ideal for
Teams whose entire value proposition genuinely is a thin wrapper on a model with no proprietary data or workflow to defend
From the transcript
“you need to make sure that your product gets better as the llms get better”
“too many people are building things to make up and compensate for the llms that all that work is going to go away”
“your differentiator is something that will continue to be there as the llms get better and smarter”
“we need to build those things to guard rails to help with the change to help suggest”
“an Enterprise CIO will go use chaty on the weekends but they come to work and they expect their software to be deterministic”
From the episode
Lessons in product leadership and AI strategy from Glean, Google, Amazon, and Slack
Tamar Yehoshua (Product at Glean, ex-Google and Slack)