An AI evaluation is a production operating control

What the paper develops

Consider a workflow that passes a careful launch test, then falls behind the work it was built to handle. Later, the vendor updates its model, another team changes the input format, and the business adds a case type the test never covered. Each change seems reasonable alone. If no one checks the combined effect, poor results may go unnoticed until a complaint reaches the team.

The launch test was sound, but its result grew old as the system and the work changed. A production evaluation must keep checking today's cases and give the workflow owner power to limit use when the evidence no longer fits.

A benchmark compares models on common tasks, while a red team searches for ways to make one fail. A production evaluation asks whether this team can depend on this system for its current cases. Each result covers only what its test checked, so the team should not use one test to answer another question.

Seven jobs keep that answer current. Choose cases, set limits, keep the tests distinct, watch live work, retest after change, learn from failures, and give someone authority to act.

Build true-to-life cases

Choose cases from real work, including messy inputs and rare errors that would cost the team dearly. Record where those cases came from, then limit use to the kinds of work they represent.

Set thresholds by consequence

A high average can hide failure in a critical case type, so set a limit for each type before testing. Make the limit stricter where an error costs more or where a person must review the answer.

Monitor production signals

Watch for new inputs, more human corrections, rising exceptions, and reviewers who disagree about the same cases. Give each warning a clear trigger and an owner who must respond when it fires.

Turn change into a test event

A model update, prompt edit, or new data source may change answers the team once trusted. Retest the cases that change could affect before people depend on the new version.

Learn from incidents and near misses

After a failure or near miss, look for its cause across the model, data, instructions, and review process. Add a case that would catch it next time, then check whether the fix holds.

Assign authority to change allowed use

Name the workflow owner who can limit, pause, or restore use as the evidence changes. A warning becomes a control only when it reaches someone with the power to decide.

Run it as one system

Cases give the limits meaning; live warnings and system changes tell the team when to retest. Incidents improve the cases, and the owner uses those results to decide how much the workflow may depend on the system.

A passing result supports only the kinds of cases and circumstances the evaluation actually covered. It cannot promise that the system will be safe in every future setting, and it should never approve new case types by analogy. When a model version, source, or workflow changes, the owner must decide whether the existing cases still represent the work and whether allowed use must narrow while the team retests.

The cost of this care should match the stakes. A low-impact workflow may need a modest sample and periodic review, while a workflow people depend on every day needs more frequent checks and a practiced way to stop.

Compare the date of the last production test with the last material change to the model, data, or workflow. If the change is newer, ask what that old test still covers, then retest or limit use until the evidence catches up.

Build the next test from real work, set limits before seeing results, and give each live warning an owner. Fund the time to keep those checks current. The white paper shows how the seven jobs connect and how to match the effort to the risk.