Give It Your Hardest Task
Evaluate a serious AI tool on your gnarliest real problem, not a dumbed-down toy.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 3
- Confidence
- 85%
Counterintuitively, the right way to test a professional-grade coding agent is to hand it your hardest real task rather than a trivial vibe-coding demo. A capable agent is built to absorb genuinely hard, bounded problems; toy tasks both undersell it and fail to teach you how to use it. The task should be hard but still a single question or unit of work.
Origin
Alexander Embiricos offered this as the adoption heuristic for Codex, contrasting it with the 'start easy' approach many people take with other coding tools.
Core principles
- 01Professional tools reveal their value on hard problems, not easy ones
- 02A dumbed-down test task hides the capability you're trying to measure
- 03Hard-but-bounded beats hard-but-sprawling — scope matters
- 04Building up to bigger tasks over time is how you learn to prompt it
How to run it
- 1
Pick a real, hard problem you actually have
Choose a genuine open problem — e.g. a bug you can't find the cause of, or a fix you don't know how to implement — rather than inventing a trivial exercise.
Pro tip Karpathy's example: feed it the gnarliest bugs that have eaten hours; let it run and it can solve what nothing else has.
- 2
Keep it to a single task or question
Constrain the test to one bounded unit of work so you can actually judge the result, instead of an open-ended mega-goal.
Watch out Don't hand it 'build a new unicorn business' — that's not one task and it won't work yet.
- 3
Build up to bigger things as trust grows
Once you've seen it handle a hard bounded task, expand toward larger, multi-step, longer-running work.
Pro tip Run a few things in parallel — a hard task, a code-base understanding pass, and a plan you co-write — to learn its range fast.
In the wild
Karpathy reported giving Codex the gnarliest bugs he'd spent hours on that nothing else had solved, letting it run for about an hour, and getting a fix.
→ Codex earned a reputation for tackling the hardest bugs, which became a signature use case cited in its growth.
Common mistakes
Testing with a trivial or random task
Dumbing the task down to something trivial hides exactly the deep-reasoning capability that distinguishes a professional agent, so you underrate the tool and never learn how to drive it.
Handing it an unbounded mega-goal
Asking for a whole business or an entire product in one shot exceeds what the agent can do today and produces a failure that misleads you, when a hard single task would have succeeded.
Is it for you?
Best for
Technical users deciding whether to adopt a serious coding agent, who have real hard problems on hand
Not ideal for
Casual exploration where you only want to kick the tires on a throwaway idea
From the transcript
“the best way to try Codex is to give it your hardest tasks, which is a little different than some of the other coding agents”
“I think it's like give it like the hardest problem, but something that is still like one like question”
“the hardest problem I have is that I need to build like a new unicorn business." Like, obviously, that, you know, it's not going to…”
From the episode
Why humans are AI’s biggest bottleneck (and what’s coming in 2026)
Alexander Embiricos (OpenAI Codex Product Lead)