Autonomous code agents run circles around human workflows in terminals, repositories, and CI/CD pipelines because they interact directly with machine-readable project files and deterministic tools. Yet the moment industry attempts to deploy that same autonomy across graphical desktop software, progress hits a wall. Most computer-use architectures remain stubborn hostages to multimodal vision models that capture pixel screenshots and simulate clumsy mouse clicks.
Researchers Rui Xie and Lu Chen from the X-LANCE Lab at Shanghai Jiao Tong University and BIGAI argue that visual grounding on raw pixels is an inherently fragile foundation for autonomous execution. To bypass this bottleneck, the team introduced ASIL (Agent-Software Interaction Layer)—an agent-native interface designed to replace expensive pixel parsing with structured JSON observations and executable semantic code.
The Failure Modes of Visual UI Control
Current computer-use models force neural networks to inspect rendered displays, estimate coordinate offsets, and emit low-level motor primitives such as cursor drags, clicks, and keystrokes. In their benchmark paper, the researchers demonstrate why screenshots fail fundamentally: a rendered bitmap represents only a lossy visual projection of an interface. A single flat image obscures inactive panels, background processes, metadata trees, and document hierarchies, all while forcing the model into redundant, compute-heavy multimodal inference steps over cosmetic layout changes.
Because GUI events rely on coordinate-sensitive motor primitives, a straightforward user intention bloats into long, error-prone step sequences. A minor padding tweak or visual theme shift shatters coordinate predictions, derailing multi-step planning loops. ASIL bypasses the presentation layer entirely by extracting structured JSON state representations and executing tasks through direct file-, script-, and service-level access paths across applications.
Empirical Performance and Token Efficiency
The research evaluated ASIL across 15 software environments against a benchmark of 300 single-application and 80 multi-application tasks. Paired with closed foundation models, ASIL cleared an 80% execution success rate while requiring fewer than five discrete actions per completed task.
Under a repaired runtime and a 50-step budget, traditional screenshot-and-click control scraped together just 6.6% and 26.6% strict success rates on the same task sets, rising to 15.0% and 53.3% on an easier OSWorld-comparable baseline. Against application-native programmatic interfaces on matched tasks, ASIL surpassed LibreOffice's UNO API by 28 to 38 percentage points while matching draw.io's MCP content contract. Structured state representations also streamline open-weight model optimization: applying supervised fine-tuning lifted Qwen2.5-2B performance from 58.0% to 72.1% and Qwen2.5-9B from 66.6% to 80.4%, with lightweight on-policy reinforcement learning pushing scores to 74.4% and 82.2%.
The strategic takeaway for enterprise architecture is clear: dependable workflow automation requires structured state exposure and semantic API contracts, not incremental improvements in visual grounding models. While ASIL proves that eliminating pixel parsing slashes action overhead and stabilizes success rates, its broad enterprise deployment remains gated by legacy desktop software lacking exposed scripting hooks or standardized file interfaces—highlighting that the ultimate challenge in agentic automation is interface design, not vision scale.