We have reached a point where the tech giants' fight for leadership in AI agent benchmarks comes down not to neural network architecture, but to basic hardware power. Infrastructure noise and environment constraints quietly distort test results, turning objective metrics into a contest of compute budgets.
What benchmarks actually measure
As research data indicates, the market prefers to ignore the problem of infrastructure noise in benchmarks for AI agents that write and execute code. Imagine two agents running on the exact same language model showing a three-percentage-point performance gap. It looks like the superior algorithm is winning. In practice, the reality is simpler and more expensive: one agent operated under strict limits, while the other received more resources.
In the test, the performance gap between the strictest and unrestricted configurations reached six percentage points.
As running the test in a live environment revealed, the results did not match official leaderboards, and a chunk of tasks failed due to pod errors entirely unrelated to model capabilities. Starting from a certain version, the benchmark specifies recommended CPUs and RAM for each task, but a massive gap in stability lies between a recommendation and a hard limit.
The price of hardware headroom
The mechanics are straightforward: a container has a guaranteed resource allocation and a hard ceiling, past which the system kills the process with an OOM error. Without sufficient headroom, a momentary memory spike terminates the container. To understand the scale of the issue, the team ran the benchmark across configurations ranging from strict to entirely unrestricted limits.
Infrastructure error rates dropped notably when moving from strict to unrestricted configurations, yet overall success rates only shifted within the margin of statistical error. Basic throttling consumed resources without making the model any smarter. Past a certain threshold, however, the dynamic changed radically.
The final delta between extreme points: +6% in success rate, while the share of infrastructure errors dropped from 5.8% to 0.5%.
As researchers note, this effect reproduces across other models, confirming that the issue is systemic. Verification on a secondary benchmark showed identical dynamics on multi-run tasks up to a specific memory multiplier over baseline, where the performance variance hit 1.54%. Businesses are overpaying by selecting systems based on "paper" benchmarks without factoring hidden deployment infrastructure requirements into the budget.