The market for long-running autonomous systems has finally moved past breathless press releases straight into a brutal efficiency audit. The core problem is stubborn: when algorithms try to chain together multi-step business logic, they hit a hard wall of basic incompetence. Researchers from Microsoft and the City University of Hong Kong built a specialized testing framework precisely to measure how well AI agents make strategic decisions during extended workflows. Existing benchmarks merely look at whether an agent eventually finishes a task, completely ignoring the structural rot of its intermediate steps.
Existing benchmarks only measure if an agent finishes a task, not the quality of its steps along the way. Taste-Bench is designed to fill that gap.
Underpinning this new metric are tracked agent execution traces across software engineering and research benchmarks, including SWE-bench Pro and public transcripts of METR's MALT dataset. The resulting corpus packs 502 distinct decision points, feeding models a base task, a specific execution trajectory up to a fork in the road, and two possible pathways forward.
Financial Risks and Accuracy Decay on Long Distances
The economics of deploying autonomous agents run straight into a systemic trap: early mistakes masquerade as clean, functional code while burning through enterprise compute budgets long before anyone spots the failure. Testing a lineup of models from Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral, the researchers found that even the top performers barely clear the sixty-percent threshold. The leading contender, GPT-5.6 Sol, managed an average of just 59.7 percent, with GPT-5.5 right behind at 59.5 percent.
The best model tested, GPT-5.6 Sol, averaged 59.7 percent, closely followed by GPT-5.5 at 59.5 percent.
When a crucial clue sits squarely in previous steps, model accuracy climbs to a modest 62.3 percent. But when that hint is delayed or obscured, accuracy crashes down to 21 percent—statistically indistinguishable from pure guesswork. Throwing more compute at the reasoning phase fails to fix the underlying rot, as models simply waste expensive tokens on branches where the necessary data physically does not exist yet in the context window.
The Infrastructure Trap and Budget Reallocation
Scaling autonomous systems hits a brick wall because models simply cannot forecast the downstream consequences of early architectural choices without constant human babysitting. The new dataset demonstrates that even deep testing loops fail to prevent catastrophic logic drift on long horizons, turning automated workflows into an expensive line item of operational waste. Engineering leads must immediately audit their API spend limits and vendor contracts to account for the hidden compute tax inflicted by flawed code branching.