Should an AI reviewer give a score or a verdict?
A verdict. Yes or no holds steadier than zero to a hundred.
- Hypothesis
- An AI reviewer judges more reliably when it answers a clear yes-or-no question per criterion than when it rates the whole change on a scale.
- Method
- A fixed set of test specs with known expected outcomes, held constant between runs. We ran the same reviewer in two forms, once rating the change on a zero-to-a-hundred scale and once answering a yes-or-no question for each criterion, and compared how often each one matched the expected outcome.
- Finding
- On a scale, the same unchanged code earned visibly different scores from one run to the next. With yes-or-no questions, agreement with the expected outcome rose from roughly four in five to nearly every case, and repeated runs stayed put.
A score feels precise. 78 out of 100 looks like a measurement. Run the same review again on the same code and you may get 85. Nothing changed except the dice.
We saw exactly that early on, so we put it on the bench: the same specs, the same expected outcomes, the same model. One version of the reviewer rated each change on a scale. The other answered a plain question for every criterion: is it met, yes or no, and why not.
The yes-or-no version won clearly. It matched the expected outcome far more often, and it gave the same answer when we asked twice.
Why it works
A scale invites the model to weigh everything at once and settle on a number somewhere in the middle. A yes-or-no question forces a decision about one thing, with a reason. A reason is also something the next step can act on. A number is not.
In the Factory
In the DayStorm Factory, every acceptance criterion is checked as met or not met, with a reason. When a criterion is not met, that reason goes straight back to the build, so the fix targets what is actually missing.