Regression blindness: self-improving agents have a problem

One of the more interesting findings in Agentic Harness Engineering, from Jiahang Lin, Shichun Liu, Chengjun Pan and collaborators, is not that agents can improve their own harnesses.

It’s that they are much better at predicting what a change will fix than what it will break.

Before each harness modification, the system predicts which tasks should improve and which might regress. The authors find that fix prediction is reasonably strong, but regression prediction is much weaker -> a failure mode they call “regression blindness.”

That matters because harness improvements are not additive.

Several components in the paper improved performance individually. But when combined, the total gain was smaller than you would expect from adding those improvements together. In some cases, a single component performed better than the full evolved harness.

Why? Because the changes interacted. Memory, middleware, prompts, and verification logic can all consume the same limited task budget. Add too many “good” mechanisms and the agent can spend more time checking itself than solving the task.

I think this gets at a deeper problem in self-learning systems.

It’s easy to imagine an agent accumulating successful lessons over time:

• this worked → keep it
• this worked too → add it
• this also worked → add it

But eventually those lessons begin to interfere with one another. So the hard problem may not be getting an agent to improve itself. It may be understanding the second-order effects of improvement.

For enterprise agents, that means the evaluation system may be just as important as the learning system. You need to know not only whether a new version is better on average, but whether fixing one class of errors quietly created another. Which opens up another related problem: over-fitting. But we’ll keep that for a future post. For now, the lesson is: A self-learning agent that cannot detect regressions is not really learning safely.

Paper: Jiahang Lin, Shichun Liu, Chengjun Pan et al., Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (2026).