Production Drift measures live answers against the same rubric the tests used. The person sees quality over time, not only at release. A slip shows up as a change against the standard, with the cases behind it. Failures found here become new test cases.
When to Use It
The prompt runs in production at volume
Inputs change faster than the eval set
A slip would be felt by real humans before anyone looked
When Not To
Volume is too low to see a trend
Scoring would send private data to a model with no plan for it
No rubric exists; production scoring without a standard is a dashboard
Three views
The exchange between the human, the agent and the system the agent acts on; who holds each part of it; and the component that implements it.
01 · the interaction
02 · who holds what
01SampleWhich live answers are scored, and how many
02Same rubricThe standard the tests used, applied to live traffic
03TrendQuality over time, by dimension
04AlertA slip past a threshold reaches the person
05Back to the setA production failure becomes a test case
03 · the component
production drift · primitives
Sample
Same rubric
Trend
Alert
Back to the set
primitive wireframe, generated from the anatomy — the installable component ships when this
pattern's anatomy stabilizes
The release decision on a live agent used to be a guess. Here it is a scored comparison against 140M real past conversations. That changes what a product team owes the reviewer before shipping.
The brief you write for an agent shapes its behavior only where that behavior can be checked. Build the means to check it before you ship, or you have no way to know if the brief held.
Three words on the release check beat a dashboard that goes unopened. The open question is who decides: failing the job on a regression is a setting, and a person still owns the call.
Writing the test set was the chore that kept teams iterating by feel. Describing the failure in a sentence and getting rows and a scorer back is the step that turns the chore into a habit.
Ten at a time is the right unit. It is enough to calibrate a judge and small enough that a designer will do it over coffee. That is the difference between a habit and a study.
Two standards: if production is scored by a different rule than the tests, a slip in one is invisible in the other. One rubric, two places.
Sampling cost: scoring each live answer with a model is expensive. The sample must be enough to see a trend and no more.
Alert fatigue: a threshold set too tight pages humans for noise. Set it from the run-to-run variation the tests showed.
What we are learning from the community
Readers who build with Production Drift add what they found: questions, use cases, critiques, other ways to solve it.
An editor reads each one before it shows here. Nothing here changes the pattern's status on its own.
This pattern has no contributions yet. The form below sends the first one.
Sep 11, 2026Added to the Watching list from the prompt-evals brief and the 2026-09-11 vendor pass; the Chromatic review loop shapes the comparison and release entries.