Communal AI Experimentation Loop
Discover valuable AI use cases by experimenting visibly with other people.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 94%
Frame heavy AI use as an investment in experimentation rather than as a goal to maximize token consumption. Select one or two meaningful problems, use the technology directly, and expose the process to colleagues through a shared channel or working session. Participants publish prompts, outputs, failures, and partial discoveries so others can try nearby variations. A weak initial idea can become valuable after several people iterate on it, allowing communal discovery to compound faster than isolated exploration. Once a pattern repeatedly solves a real pain point, take it beyond tinkering by shipping an end-to-end workflow and gathering user feedback. This approach also makes adoption more joyful: people who are uncertain or fatigued can pair with an enthusiastic practitioner instead of having to invent the perfect use case alone. Depth and shared learning matter more than breadth or raw spending.
Origin
Dianne Penn described Anthropic employees testing early Claude versions together in a public internal Slack channel, where shared ideas and variations sometimes produced a new use case within roughly ten requests.
Core principles
- 01Optimize token use for learning and useful outcomes, not consumption itself.
- 02Touch the technology before forming a fixed strategy.
- 03Experiment in public so discoveries can compound.
- 04Let peers vary promising ideas instead of starting from zero.
- 05Go deep on one or two valuable problems rather than sampling everything.
- 06Treat experimentation as a communal activity rather than an individual burden.
How to run it
- 1
Define the learning outcome
Choose a meaningful problem or capability to explore and state what useful learning would look like. Treat tokens and time as inputs to that outcome.
- 2
Touch the technology
Use the current model or prototype directly before producing a fixed strategy. Explore both successful and failed interactions.
- 3
Work in public
Post experiments, prompts, outputs, and observations in a channel visible to the group. Make partial discoveries easy for others to understand and reproduce.
- 4
Vary promising ideas
Invite other participants to modify the prompt, context, tool access, or workflow around an interesting result. Let multiple perspectives search the nearby possibility space.
- 5
Identify repeated value
Look for a pattern that solves a specific user pain point reliably enough to deserve deeper investment. Separate novelty from durable usefulness.
- 6
Ship and learn
Build the strongest discovery into an end-to-end workflow, expose it to users, and collect feedback. Feed the resulting observations back into the shared experimentation loop.
In the wild
When Anthropic had fewer product surfaces, nearly the entire company tested early Claude versions in a shared Slack channel. Employees tried tasks such as editing essays and drafting emails, while colleagues copied promising ideas and tested variations. Penn said that within roughly ten requests, these communal iterations could reveal something magical or expose a potential new use case.
→ Shared experimentation accelerated use-case discovery and helped build a bottom-up culture around Claude.
A manager who finds AI tools frustrating pairs with a colleague who enjoys experimentation. They select one recurring reporting problem, share each attempt in a team channel, and invite two coworkers to improve the workflow. After several variations, they produce a reliable draft-and-review process and deploy it for the next reporting cycle.
→ The manager gains a useful workflow and practical confidence without having to explore dozens of unrelated tools alone.
Common mistakes
Maximizing spend instead of learning
Tokens are an input, not the outcome. Define the experiment, user value, or workflow improvement the spending is meant to produce.
Experimenting entirely alone
Private exploration prevents promising ideas from benefiting from other people's variations and makes adoption feel more burdensome.
Sampling too many tools shallowly
Constantly switching prototypes can leave users with many broken half-workflows. Go deep on one or two valuable problems.
Is it for you?
Best for
It is best for teams evaluating new models, agent products, or prototypes whose valuable use cases are not yet obvious.
Not ideal for
It is not ideal when experimentation would expose confidential data or when a stable, well-understood workflow simply needs disciplined implementation.
From the transcript
“It's almost like token spin is more the input, and really the output is what you described of experimentation.”
“It's very hard to come up with a perfect strategy without touching the technology when it's moving this quickly.”
“Like experimentation is not always necessarily an individual sport.”
From the episode
Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future