Finding High-Leverage AI Ideas
Give AI work a metric, run hackathons, and study what makes AI products feel magical.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 3
- Confidence
- 88%
A three-part system for generating genuinely good AI ideas inside a company, instead of chasing 'AI for everything.' Give AI experimentation its own metric (how many shots you take), run hackathons to make the tech approachable and surface which ideas stick, and obsess over the UX details of AI products that already feel magical to learn what to emulate.
Origin
Aman Khan's practical playbook, developed running experimentation and hackathons at Arize AI.
Core principles
- 01Every AI experiment needs a metric even when it won't move revenue yet
- 02The right early metric is how many shots you're taking
- 03Hands-on hackathons remove the feeling that AI is unapproachable
- 04Most AI ideas will fail on contact — that's information, not defeat
- 05Study successful, magical AI products to learn what 'good' looks like
How to run it
- 1
Give AI work a metric — count your shots
PMs are used to moving a business metric, but prototyping with AI usually has none, so teams claim it 'isn't moving revenue' and can't tell if it's working. Instead, measure how many shots you're taking — the volume of experiments.
Pro tip A shot-count metric legitimizes exploration that isn't yet tied to revenue.
Watch out Without any metric you can't tell whether your AI efforts are working at all.
- 2
Run hackathons to surface where ideas stick
Get everyone hands-on to remove the sense that AI is unapproachable. Bring a list of real problems to solve, try applying AI to each, and watch which ideas stick and which fail. Expect maybe 10 problems where 9 of the AI attempts don't work well.
Pro tip Come with the problems in hand so the hackathon tests real needs, not toy demos.
Watch out Some hard-looking use cases fail on hidden context — Arize's on-call Slackbot broke because who-owns-what context kept shifting between people and teams.
- 3
Obsess over the details of magical AI products
Study successful AI products and ask what they get right. The goal is to find experiences where AI is actually magical and reverse-engineer the UX details that make them work.
Pro tip Do live teardowns — Arize live-streams webinars dissecting a cutting-edge AI product (e.g. NotebookLM) every week or two.
In the wild
At an Arize hackathon, the engineering team built a Slackbot to alert the right on-call person when someone posted a problem in a support channel — classify the problem, ping the owner. It seemed like a perfect, simple AI use case, but turned out to be very hard because so much context was missing: the person who owned a problem two weeks ago had moved to another project and a different team had taken over.
→ A vivid lesson that identifying the right problems to solve — not the AI technique — is the real difficulty, and that hackathons surface this cheaply.
Arize did a live-streamed teardown of the NotebookLM product, planning to repeat the format every week or two on a cutting-edge AI product to understand the space and how great AI UX works.
→ A repeatable ritual that builds team intuition for what makes AI products magical.
Common mistakes
Running AI experiments with no metric
Teams say AI 'isn't expected to move a business metric,' then have no way to judge if the work is succeeding — so effort drifts and can't be defended or improved.
Underestimating hidden context in 'easy' use cases
Problems that look trivial for AI (like routing a support question to the right owner) fail because critical context — ownership changes, project handoffs — isn't available to the model, wasting build effort if not caught early via a hackathon.
Is it for you?
Best for
AI PMs and product leaders whose teams are flooded with undifferentiated 'let's add AI' proposals
Not ideal for
Teams with a single validated AI bet already in execution that don't need idea generation
From the transcript
“every aipm actually needs a metric and every PM within a business is tasked with moving some business metric but there actually isn't one for…”
“how many shots are you taking in the first place”
“I think hackathons are great I think that getting people to be Hands-On in the first place it really removes that feeling that this this…”
“the goal is really to find experiences where AI is actually magical and what do they get right”
From the episode
Becoming an AI PM
Aman Khan (Arize AI, ex-Spotify, Apple, Cruise)