Eval Run Setup is the moment before a test runs. The person picks the prompt versions, the models and the example set to test them against. The tool says what the run will cost and how long it will take before the person starts it. The run then carries a record of what was tested.
When to Use It
A run costs real money or real minutes
More than one version or model is in play
The team needs a record of what was tested
When Not To
Runs are free and instant; just run them
There is one version and one model; the choice is already made
The estimate would be a guess with no basis
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01Prompt versionsThe versions to test, at least a baseline and a candidate
02ModelsThe models each version runs on
03Example setThe eval set the run uses, at a named version
04EstimateCost and time for the run, shown before the person commits
05Run recordWhat was tested, when, by whom, at what cost
03 · the component
eval run setup · primitives
Prompt versions
Modelsoncesessionalways
Example setEstimateRun record
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
The harness is the signal the reviewer depends on. If it's too expensive to run or too noisy to trust, the human's review step loses its foundation before it begins.
A threshold is a decision made once, by whoever wrote the workflow file. The product does not ask whether this regression should block or only warn; that question needs a screen, not a setting.
A vendor retiring the surface moves the interaction into the team's own tools. That is an argument for patterns that outlive any one product, and for baselines the team owns.
Estimate honesty: a cost figure that is often wrong trains humans to ignore it. Show the range and the assumptions behind it.
Grid explosion: three versions times four models times two sets is a lot of runs. Defaults should make the smallest useful comparison the easy path.
Reruns: outputs vary run to run. One run of a small set can flip a verdict; the setup should say how many repeats it used.
What we are learning from the community
Readers who build with Eval Run Setup add what they found: questions, use cases, critiques, other ways to solve it.
An editor reads each one before it shows here. Nothing here changes the pattern's status on its own.
This pattern has no contributions yet. The form below sends the first one.
The Story So Far
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.