I built the instrument to find out. Its most valuable output was telling me, repeatedly, that I was wrong.
Ask a team how they know their AI output is good and the honest answer is almost always: we look at it. That fails in three specific ways, and I hit all three.
I was building a pipeline that turns written material into narrated video with generated imagery. Quality was subjective, failure modes were subtle, and I had no way to tell whether a change helped. So before improving the output, I built something to measure it.
The critic never sees the source — only the finished rendered output, exactly as a stranger would encounter it.
A judge that has read the script will confirm the video makes sense. Because it does — to someone who read the script.
The judge watches and retells what it understood. Comparing that against what the piece was meant to convey shows exactly where comprehension broke: which scene, which transition, which unexplained object. A score says the output is bad. A retelling says why, and points at the line to change.
Known-good and known-bad material goes through the judge to verify it can tell them apart. Without this an evaluation harness is a random number generator with a confident tone — and I would have trusted it.
An evaluation harness only earns its cost when it contradicts the person who built it.
I reported that character identity was holding across generated frames — the contact sheet looked fine. Judged at full resolution, the same sequence contained three visibly different people playing one character. At thumbnail size the differences were invisible.
Looking at it is not an instrument.
I proposed replacing a working approach with one I thought was cleaner. Calibrated head to head, it lost. I killed it the same day I recommended it.
Without the harness I would have shipped it on the strength of my own enthusiasm, then spent weeks wondering why output had got worse.
A change meant to correct unnatural character posing dropped scores by eight points. I reverted the harmful half and kept the rest. Unmeasured, that reads as “still not fixed, try harder.”
I reported that quality tracked an input variable monotonically, based on three samples. At five samples the relationship broke. Three points will support almost any story you want to tell.
A piece the harness scored well read, to a human, like a dream — abstract, elements unconnected, technically fluent and semantically empty. The harness had no axis for that. It measured whether each part was intelligible, not whether the parts cohered. I added coherence as a separate heavily-weighted axis and re-ran everything. Scores that had looked strong dropped sharply, which was correct.
An evaluation harness is itself software with bugs, and the worst bug is a missing dimension. It never reports an error. It reports a good score for work that fails in a way you forgot to measure.
| Measure | Before | After |
|---|---|---|
| Comprehension (blind judge, /10) | 3.3 | 7.8 |
| Overall (mean, /140) | — | 103.8 |
| Judge agreement (two runs) | — | r = 0.78 |
| Judge drift (mean abs. diff) | — | 0.27 |
Those last two rows matter as much as the first: I measured whether the measuring device agreed with itself before trusting what it said.
All 56 cycles are documented, including every dead end. Several were expensive. The record is what stops me re-running them.
Any team with a generative feature in production and no automated notion of output quality — which, in my experience, is most of them. Concretely: LLM output regression testing before release, A/B evaluation of prompt and model changes, detecting quality drift after a model upgrade, and building review that scales past what humans can read.
Also directly relevant where this has to run locally, because the content being judged cannot be sent to a third-party API.
Private Call Analysis Pipeline · Self-Healing Agent Fleet · All work