Principles for Building Trustworthy AI Products
Match UI confidence to data quality, be transparent about sources, and design virtuous data cycles.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 3
- Confidence
- 80%
From 15 years building AI/ML products (Google Knowledge Graph, Foursquare recommendations, Slack search), Weiss distills principles for shipping AI that earns trust. The promise implied by the UI must match the quality of the underlying data — a core failing of current LLMs, which appear supremely confident even while hallucinating. Add transparency about where data comes from to build credibility, and design virtuous cycles where using the product generates training data that makes the model smarter over time.
Origin
Synthesized by Noah Weiss from working on Google's Knowledge Graph (~2008), Foursquare recommendations, and Slack search/ML. He cites Netflix's recommendation feedback loop as the classic virtuous-cycle example.
Core principles
- 01The confidence a UI projects must not exceed the quality of the data behind it
- 02Systems that appear certain while hallucinating undermine user trust — acknowledge uncertainty
- 03Transparency about data provenance lets users build credibility in the tool
- 04The best AI products generate training data as a byproduct of normal usage
How to run it
- 1
Match UI promise to data quality
Design the interface so the confidence it projects matches how good the underlying data actually is — don't over-promise with a polished UI on top of shaky data.
Pro tip Design the system to acknowledge when it's not sure rather than being 'so faultless', which is what preserves long-term trust.
Watch out LLMs appear supremely confident even when completely hallucinating; inheriting that false confidence in your UI will erode user trust.
- 2
Be transparent about data provenance
Show users where the data or answer comes from so they can judge and build credibility in the tool over time.
- 3
Design a virtuous data cycle
Build the product so that natural usage produces training data as a byproduct, which then makes the model smarter, more accurate, and more predictive.
Pro tip Model it on Netflix's recommendation loop, where customer feedback continuously improves prediction quality.
In the wild
Nearly 15 years ago Weiss worked on what became Google's Knowledge Graph — using NLP to extract facts from the web to answer queries like 'the tallest mountains in Europe' directly rather than returning ten blue links. A guiding principle there was that the promise of the UI has to match the quality of the underlying data.
→ The principle carried forward as a lens for evaluating today's LLMs, whose overconfident presentation Weiss flags as a trust failure product teams must design around.
Common mistakes
Shipping a confident UI on unreliable data
When the interface projects certainty the data can't back up — as LLMs do when hallucinating — it undermines the trust users have in the whole system, which is hard to rebuild.
Ignoring the data feedback loop
Building an AI product without a virtuous cycle means the model never improves from real usage, so it can't get smarter or more predictive the way Netflix's recommendation system did.
Is it for you?
Best for
Product teams integrating LLMs, ML, or recommendations into a product and worried about trust and long-term model improvement.
Not ideal for
Purely deterministic products with no ML component, or throwaway prototypes where trust and data loops are irrelevant.
From the transcript
“the promise of the UI has to match the quality of that underlying data which is to say and I think there's actually one of…”
“using a lot of transparency about where the data comes from so people can actually build credibility and the tool is really important”
“you have virtuous cycles that are naturally part of the project experience where you can get training data as a byproduct of people naturally using…”
From the episode
The 10 traits of great PMs, how AI will impact your product, and Slack’s product development process
Noah Weiss (Slack, Foursquare, Google)