The era of judging AI agents by their ability to complete one-off, isolated tasks is coming to an end. MerchantBench, a new benchmarking framework from researchers at Alibaba Group, Zhejiang University, and Peking University, introduces a brutal reality check for the e-commerce sector: a 365-day order-level simulation designed to test Long-Term Coherence. Unlike traditional metrics that cheer for immediate successes, this system evaluates an agent’s ability to maintain logic and operational discipline over a marathon-length operational cycle.
Grounded in nearly 100,000 real e-commerce records, MerchantBench forces agents to juggle 26 distinct tools across product sourcing, pricing, and cash-flow management. In this persistent environment, an agent’s early mistakes aren't just errors—they are traps that strictly constrain future strategic options. This isn't about chatty customer support; it’s about whether an AI can manage a store's survival without suffering a cognitive breakdown mid-fiscal-year.
The findings from the MerchantBench report are sobering for anyone expecting an immediate AI takeover of retail operations. Even the top-performing LLM configurations managed to reach only 27.3% of the mean net assets achieved by human participants. The data highlights a catastrophic gap in temporal reasoning: agents struggle with the inherent asymmetry of retail, where cash commitments are immediate, but fulfillment failures and market shifts arrive with a lag. As evidence accumulates, current models lose focus, drifting away from their core objectives and failing to adapt policies to the compounding complexity of a living business.
For business owners and CTOs, the takeaway is clear: conversational fluency is a vanity metric. If an agent cannot sustain an objective pursuit across mixed-latency feedback loops, it remains a liability rather than an asset. MerchantBench proves that while modern AI is excellent at talking about business, it lacks the operational stamina required to actually run one. Until these agents can master long-term coherence, they belong in the sandbox, not at the helm of a global supply chain.