Can autonomous systems evaluate their own outputs when nine totals and seven net checks pass without an external oracle to confirm task completion?
In standard enterprise deployments, agent self-improvement depends on feedback about task success, yet external feedback remains unavailable at test time. Beyond standardized benchmarks, reference answers and task completion signals vanish, forcing systems to infer success entirely from observable evidence and incomplete interaction records. Without external standards, agents face coupled operational failures: finding expected labels does not establish task completion, while generating more outputs under unchanged checks preserves the exact same blind spots.
To break this loop without retraining model backbones, a research team comprising Yitong Lin, Jianda Chen, Yihao Hu, Pan Wang, Hang Wang, Zhihao Wen, Xiaoxue Ren, and Wei Wu from Ant International and Zhejiang University introduced Verify by Doing (VbD). The architecture co-evolves Actor and Evaluator harnesses within a single task while keeping underlying model weights fixed. Co-evolving the execution and evaluation workflows addresses the operational bottleneck of blind self-reflection in isolated environments, eliminating the requirement for human supervision or external ground truth during production execution.
Decoupled Architecture and Tool-Grounded Verification
At test time, autonomous agents can either update their workflows and harnesses while keeping model weights fixed, or expend additional inference budget to generate and verify candidate outputs. Conventional tool-based critique acquires additional observations, while retrospective diagnosis can guide procedural actor updates. However, existing methods do not update the verification procedure itself, allowing systemic blind spots to misguide subsequent adaptation.
"The Evaluator actively inspects artifacts and environment state with tools; its findings ground an internal signal for output selection and guide updates to both execution and verification procedures."
Under the VbD architecture, the Actor executes assigned tasks through an editable harness containing instructions, skills, and working checks. Rather than passively reading static logs, the Evaluator runs a dedicated harness to actively inspect generated artifacts and environment state using tools, systematically recording satisfied, violated, and unresolved requirements. An Editor then reviews these findings, comparison decisions, and task history to propose structured updates to either or both harnesses.
How do these twin harnesses update without collapsing into recursive self-delusion?
Proposed Evaluators undergo rigorous comparative review on common candidates, while Actor updates require empirical support from their own fresh outputs. This mutual check ensures that evaluation criteria evolve alongside execution capabilities, contrasting with earlier systems like the Red Queen Gödel Machine, which jointly evolves agents and evaluators but relies on held-out ground truth to guide evaluator replacement.
Benchmark Performance and Inference Economics
This active, tool-grounded verification model translates into direct performance gains across enterprise automation environments. Across AutomationBench, tau3 Banking, and selected GDPval tasks, VbD achieved the highest overall scores among compared methods operating entirely without external feedback.
"VbD achieves gains of 3.3, 4.1, and 2.3 percentage points over the strongest baselines on the main backbone."
Ablation studies confirmed the direct contributions of active verification and Evaluator evolution. Selection tests from fixed candidate pools demonstrated that improved verification recovers successful outputs without requiring teams to generate new ones. In financial applications such as budget workbooks, where verifying payer assignments, consistent totals, and net revenues requires multi-step cell inspection and offline recalculation, active verification prevents false positives from corrupting downstream workflows.
Can internal evaluators co-evolving inside isolated workflows permanently eliminate blind spots without held-out ground truth, or will complex enterprise environments eventually require external validation to audit the harness updates proposed by the Editor?
Against the field
Autonomous agents once remained trapped within the blind spots of static checks or required external ground truth, but now the validation procedure evolves alongside execution during the task itself. The Verify by Doing approach shifts the heavy lifting from weight fine-tuning to active auditing through external tools in an isolated environment, securing top marks on AutomationBench without human prompts. Synchronous harness evolution breaks the loop of autonomous self-assessment without leaning on external oracles.
Our stories: Self-Improving AI Enters an Echo Chamber Without External Benchmarks (October 13).