Right Model for the Right Use Case
Route each AI feature to a model chosen for that job, not one popular LLM stretched over everything
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 92%
The default corporate AI move is to pick one fashionable large language model and force every feature into it, which quietly degrades quality across the board. GitLab instead runs around 16 models behind its GitLab Duo suite, selected per use case — and pairs this with two governing tenets: AI across the whole software development lifecycle (not just for developers), and radical transparency about which models are used, how they were trained, and what data is not used for training.
Origin
GitLab's three AI tenets, set a couple of years before this April 2024 conversation and described by CPO David DeSanto. GitLab's proprietary models came partly from its 2021/2022 acquisition of UnReview; commercial partners are Google (Google Cloud) and Anthropic, selected because they could meet GitLab's privacy requirements.
Core principles
- 01One model for every use case reduces the quality a user experiences from every AI feature.
- 02Latency tolerance differs radically by use case: block-generation can take 20-30 seconds; inline completion cannot.
- 03AI must serve the whole SDLC — only ~25% of it is writing code; making the developer 100x more effective just breaks everything around them.
- 04Transparency and privacy are AI tenets, not compliance afterthoughts: publish which models you use and how they were trained, and don't train on customer IP.
- 05Partners must meet your values, not just your feature list — GitLab could only partner with vendors that met its privacy bar.
- 06Model selection is a capability worth staffing: GitLab runs an AI model validation team of researchers.
How to run it
- 1
Enumerate the use cases before the models
Break the AI surface into distinct jobs: explaining and resolving vulnerabilities, summarizing conversations in planning, generating blocks of code, completing the next couple of lines inline, and so on. Each is a different job with different quality and latency profiles.
Watch out If you start from the model, you will retrofit use cases into it and never notice the quality you gave up.
- 2
Match a model to each job on its actual constraint
Select per use case: one model is good at explaining and resolving vulnerabilities; another is better at summarizing planning conversations. Crucially, code generation and inline code completion cannot share a model — block generation takes seconds to tens of seconds, which is unacceptable for next-line completion.
Pro tip Latency tolerance is often the deciding constraint, not raw benchmark quality.
Watch out Using one code model for both generation and completion produces a badly degraded user experience in at least one of them.
- 3
Mix proprietary, open-source and commercial models
GitLab uses GitLab-created open-source models, proprietary models from the UnReview acquisition, and commercial models from Google and Anthropic. Around 16 models total power GitLab Duo.
Pro tip Staff a model validation team — GitLab's includes researchers who have studied ethical AI and AI at scale — so selection is evidence-based rather than fashion-based.
- 4
Screen partners on privacy, not just capability
Set explicit privacy tenets — don't use customer intellectual property for training or fine-tuning; train on data that isn't the customer's — and only partner with vendors who can meet them. GitLab partnered with Google and Anthropic on this basis.
Pro tip Being trusted by more than 50% of the Fortune 100 with their IP sets the bar; the AI stack must clear the same bar the core product does.
Watch out Vendors that can't meet the privacy bar are out, regardless of how good the model is.
- 5
Publish the model list
Make the models you use, and why, publicly documented (GitLab's are on the GitLab Duo docs page). This is the transparency tenet applied to AI, and it lets customers reason about what they're trusting.
- 6
Extend AI past the developer
Build AI for product managers, QA, ops and security teams too. Only about 25% of the SDLC is writing code — 75% is everything around it, and accelerating only the developer breaks the surrounding system.
In the wild
Generating a block of code — a function — can take seconds, sometimes 20 to 30 seconds, to fully respond. A developer asking for the next couple of lines to be completed will not wait that long. GitLab routes them to different models.
→ Both experiences are usable, where a single shared model would have made one of them either slow or shallow.
GitLab Ultimate, its top tier, delivers a 7x productivity boost according to customer data analysis of their own improvements; the stated target is 10x. Customers have separately reported efficiency boosts of 50% and above from GitLab Duo.
→ DeSanto attributes the ability to keep pushing toward 10x specifically to per-use-case model selection across the ~16 models.
Common mistakes
Picking the popular model and making everything fit it
DeSanto names this as the common failure: people see a really popular LLM, commercial or open source, and force every use case into it. The result is reduced quality in every feature that model wasn't right for — invisible until users tell you.
Making the developer 100x faster and nobody else
If only ~25% of the SDLC is writing code, a 100x developer just means everything around them breaks. AI investment has to reach QA, ops, security and product or the system bottleneck simply moves.
Treating privacy as a post-hoc compliance review
GitLab made privacy one of three founding AI tenets, which then constrained which vendors it could even partner with. Bolting it on afterwards means discovering your model provider trains on customer IP after you've shipped.
Is it for you?
Best for
Product leaders building an AI feature suite across multiple distinct use cases, especially where customer IP or regulated data is involved.
Not ideal for
Small teams shipping a single AI feature, where the operational cost of running and validating multiple models outweighs the per-use-case quality gain.
From the transcript
“I encourage people to find the right model for the use case we have around 16 models we use today to make gitlab have its…”
“if you're using the same model for uh inline code completion as you are for code generation which is generating blocks of code you're going…”
“we tell you what models we're using are we're Source available so you can see how we're using them we tell you how they're trained”
“if you make your developer a hundred times more effective it just means probably everything around them is going to break”
From the episode
The GitLab way: Kindness, transparency, and short toes
David DeSanto (CPO)