GPT-6 Astra, the best of ten models tested in a new evaluation, meets about 70 percent of a task's requirements on average. Yet behind that partial completion lies an operational bottleneck: the model delivers a fully correct result in only 26.4 percent of tasks. While autonomous coding systems present near-flawless numbers on traditional benchmarks, actual implementation exposes a severe drop in reliability when handling real software modification requests.
Autonomous coding agents hit architectural ceilings on complex assignments, turning projected payroll savings into continuous overhead for manual code review. Research teams frequently observe scores above 99 percent on older benchmarks where models are permitted to run, check, and fix their own code. However, a new test named Co-Design Bench, constructed by researchers from Alibaba and several Asian universities, demonstrates that even frontier models fully resolve only about one in four real-world tasks.
The Reality Gap Behind Synthetic Benchmarks
To construct Co-Design Bench, researchers pulled assignments directly from real coding sessions where human users had to rephrase their prompts multiple times or roll back codebase changes. The benchmark tasks require agents to rework existing web applications using instructions that incorporate images, audio recordings, videos, or documents.
Evaluation relies on a two-tier verification setup: one AI judge tests the completed web application inside an active browser, and a second AI judge audits every passing verdict to detect mock-ups or hardcoded responses. Points are awarded exclusively to features that genuinely execute within the application. Under these conditions, the commercial leaderboard shifts substantially. Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max occupy the subsequent ranks, while Grok 4.6 finishes in last place.
In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.
This structural deficit explains why enterprise software teams cannot treat autonomous generation as end-to-end execution. On large-scale project overhauls, the smaller Qwen3.8-27B model managed to climb to second place, though the study authors caution against treating this isolated metric as evidence of an overall advantage across production workloads. The fundamental challenge remains systemic: model accuracy deteriorates across every tested architecture as soon as a single prompt bundles changes spanning layout, visual design, user interaction, and raw content.
The Compute Paradox and Execution Environments
Do higher token budgets and heavier inference compute eliminate these execution errors?
They do not. The data shows that GLM-5.2 consumes nearly six times as many tokens as GPT-6 Astra on smaller application tweaks while still scoring significantly worse. Pumping higher token volumes into an agent workflow fails to compensate for structural reasoning breakdowns when requirements become intertwined. The operational cost of computation escalates rapidly without producing an acceptable deployment yield.
Beyond model weights, the underlying runtime infrastructure dictates practical performance. In a dedicated test, researchers ran the same models across different harnesses, defined as the execution programs through which an agent reads, writes, and executes application code.
Qwen3.8-27B scored 75 or just 56 percent on larger overhauls depending on the environment.
This nineteen-point divergence illustrates why assessing models in isolation produces misleading operational metrics. The paper's authors recommend that organizations always evaluate the foundation model and its working harness together. At the same time, increasing tooling complexity introduces its own efficiency limits. While advanced environments can alter rank orders, a recent study on coding agents documented minimal measurable performance gains from elaborate harnesses despite those setups consuming up to three times as many tokens.
External audit data confirms that synthetic benchmark scores consistently overstate production value. A prior study by the research organization METR established that the actual acceptance rate of AI-generated code by project leads lagged roughly 24 percentage points behind reported scores on SWE-bench Verified. While Co-Design Bench omits multi-turn user interventions and conversational corrections, its single-turn results expose the true baseline of autonomous generation. Enterprise teams building workflows around these agents face a structural deficit: paying for compute that fails three-quarters of complex tasks, followed by human engineering hours spent debugging the output.
What it means
Synthetic benchmarks boast near-flawless accuracy above ninety-nine percent, yet in actual software development even the top-ranked model fully completes barely a quarter of production tasks. Companies banked on slashing payroll costs, only to discover that nearly three-quarters of generated solutions still require hands-on human review (someone has to keep broken code from wrecking production). Throwing larger inference budgets at the problem solves nothing either: GLM-5.2 burns nearly six times more tokens than GPT-6 Astra while delivering noticeably worse output. End-to-end software autonomy currently amounts to senior engineers spending their shifts cleaning up synthetic hardcoded placeholders.