Can AI grade its own work?
Not reliably. Separate the builder from the judge.
- Hypothesis
- A model reviewing its own output systematically misses its own errors.
- Method
- Past production builds with known human fixes, re-reviewed in repeated runs by an independent reviewer.
- Finding
- The independent reviewer caught most error classes consistently, without flagging clean builds.
A builder that checks its own work tends to confirm it. In the first days of our independent review stage, two of the most serious bugs it found had passed the automated acceptance check with a perfect score, with every test green. One of them was a background job that would have stopped silently in production.
So we tested the reviewer the hard way. We took builds from production where we already knew what a human had to fix afterwards, and let an independent reviewer look at them again, several times. Repeated runs matter: language models are not deterministic, and in our own data the same build reviewed several times has received different verdicts. A single run is an anecdote.
The reviewer caught most classes of error in every run, and it did not raise false alarms on builds that were clean. Separating the builder from the judge works.
In the Factory
Every build in the DayStorm Factory gets an independent senior review by a separate AI reviewer, not the builder, after its own checks, against the spec, with the power to send the change back. Passing your own criteria is necessary, not sufficient.