The Model Launch Bar
With probabilistic products, the PM — not the scientist — decides what accuracy is good enough to ship.
- Difficulty
- Moderate
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 85%
AI outputs are probabilities, not answers: a model says 'that's a rhino, and I'm 70% sure'. That means shipping is no longer a binary works/doesn't-work call — someone has to decide where the accuracy bar sits for real users. Nika insists that decision is the PM's, and pairs it with a second PM-owned decision: whether to use an off-the-shelf model or invest in proprietary data, because everyone buying the same ready-made dataset ships the same quality.
Origin
Marily Nika's framing of what makes the AI PM role distinct, from shipping ML products at Google and Meta.
Core principles
- 01A model takes an input and outputs a prediction plus a confidence — never a certainty.
- 02Where the quality bar sits is a product decision about user tolerance, not a technical one.
- 03If everyone trains on the same purchasable dataset, everyone's quality is identical — differentiation comes from your own data.
- 04The PM must also own the monetisation bridge from research output to a paid use case.
How to run it
- 1
Define the user-facing failure mode
Work out what happens to the user when the model is wrong. A wrong movie recommendation and a wrong medical flag have wildly different tolerances.
- 2
Set the accuracy bar explicitly, and own it
Decide the number: is 70% accurate good enough? 80%? State the bar before training finishes so the team isn't negotiating it against sunk cost.
Pro tip Nika treats this as the fun part of the role — it's a decision nobody has tackled before, and it's entirely yours.
Watch out Do not delegate this to the data scientist; they optimise the metric, they don't own the user's tolerance for being wrong.
- 3
Decide off-the-shelf vs. proprietary data
If you're a large company offering a service where quality is the differentiator, train on more and more diverse data of your own. Buying a ready-made dataset from an agency means shipping the same quality as everyone else who bought it.
Watch out Agencies sell the same data packages to your competitors — parity, not advantage.
- 4
Bridge research to a monetisable use case
Take the research scientist's amazing idea and figure out how it actually monetises — willingness-to-pay research (what's the minimum you'd pay, the maximum, what would you want if you paid) is the PM's job, not the scientist's.
In the wild
Nika's recurring illustration: the classifier says it's 70% sure a photo is a cat. Is 70% good enough to put in front of users, or is the bar 80%? Nobody but the PM can answer that.
→ The accuracy threshold becomes an explicit, defensible product decision rather than a number the data scientist happened to land on.
Nika points to the sign-up form circulating in early 2023 asking users what the minimum they'd pay would be, what the maximum, and what they'd want to see if they paid — an example of a PM bridging free research capability to a priced product.
→ A paid tier launched shortly after, priced from that demand signal rather than from cost.
Common mistakes
Treating model accuracy as an engineering metric
Accuracy targets set without a view on user tolerance either block a shippable product forever or ship something users don't trust. The bar is a product judgment.
Buying the same dataset as your competitors and expecting an edge
If everyone takes the exact same ready-made dataset, every company's model quality converges. Diversifying and collecting your own data is the only route to a quality advantage.
Is it for you?
Best for
PMs shipping their first probabilistic feature and deciding when it's good enough for users.
Not ideal for
Safety-critical domains where the acceptable error rate is set by regulation, not by product judgment.
From the transcript
“it's totally your responsibility as a p to decide okay the recognition of whether this phold is a cat or a doob is good enough…”
“there are agencies that are selling data packages of data that are ready so you can get them and train your models but the question…”
“if you're a PM you need come up with ways to actually”
From the episode
AI and product management
Marily Nika (Meta, Google)