Do more rules make a better coding agent?
No. Focus beats a longer checklist.
- Hypothesis
- Adding more rules to an agent's instructions does not automatically improve its work, and can make it worse on complex tasks.
- Method
- The same specs at three levels of difficulty, the same model, built twice. Once with the build instructions in use at the time, once with an extended version carrying about a third more text, mostly additional rules for layout, dependencies between files and patterns to avoid. Each result was checked against the same criteria.
- Finding
- On the easy and the medium task, both versions delivered the same result. On the hard task, which touched many files, the extended version was faster but left several files unchanged that the change needed. A similar test on the instructions for repairing failed builds pointed the same way. More structure did not mean better repairs.
When an agent makes a mistake, the reflex is to add a rule. Don’t forget the layout. Update every dependent file. Avoid this pattern. Each rule makes sense on its own.
We tested what happens when they pile up. Same tasks, same model, two sets of instructions: the one we used, and one with about a third more rules.
On simple tasks it made no difference. On the hardest task, the one spread across many files, the longer instructions made things worse. The agent moved faster and stopped early. Several files the change depended on stayed untouched.
Why it happens
Instructions compete for attention with the task itself. Every general rule is something the agent weighs, whether it applies or not. On a hard change, the spec is what matters most, and a long checklist pushes it into the background.
In the Factory
In the DayStorm Factory, the spec stays at the center of every build. Changes to the Factory’s own instructions are measured against fixed tasks with known outcomes before they go in, and a change that does not beat what is there stays out.