Cosmetic code changes consistently break autonomous AI coding agents, exposing a critical gap between leaderboard vanity metrics and production reliability. According to research from Carnegie Mellon University, Microsoft, UIUC, and Colorado State University, applying basic semantics-preserving transformations (SPTs)—such as control-flow restructuring, dead-code injection, and identifier renaming—triggers a mean resolve-rate plunge of up to 6.7 percentage points in affected setups.
The researchers benchmarked two agentic scaffolds, mini-SWE-agent and OpenCode, backed by models including Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B across SWE-bench Verified and SWE-bench Pro. Performance degraded significantly in 6 out of 16 tested configurations. Notably, leaner scaffolds like mini-SWE-agent proved far more resilient to structural perturbations than complex architectures, proving that compounding prompt engineering often amplifies brittle failure modes.
Crucially, agent robustness follows an unpredictable jagged frontier rather than a dependable model hierarchy. While Qwen 3.6-27B ranked among the most stable models under mini-SWE-agent on SWE-bench Verified, it collapsed as the most brittle under OpenCode. For engineering leaders and CTOs eyeing autonomous developer agents, the takeaway is stark: public benchmark leaderboards like SWE-bench fail to capture how models behave under ordinary codebase churn, making strict human-in-the-loop oversight non-negotiable before deploying agentic workflows into enterprise pipelines.