Specific Labs introduced the Real-SWE benchmark in September 2026 to measure how frontier AI models handle private, real-world enterprise codebases. Rather than testing systems on public datasets, the benchmark licenses actual tasks directly from production codebases owned by operational companies. In these environments, software engineering agents face proprietary systems where source code, internal logic, and prior solutions cannot be retrieved from public internet archives.
The benchmark focuses on engineering problems that carry direct operational consequences across integrated systems. Tasks involve managing billing cycles, calculating dynamic sales taxes, and migrating active customer accounts across multiple infrastructure services.
"Expert-generated or synthetic tasks can be well designed, but they aren't the verbatim, actual tasks that engineers in real companies need to do."
As Specific Labs explained regarding the benchmark design, evaluated tasks differ from synthetic equivalents on two core axes: the underlying coding artifact and the specificity of the instruction provided to the agent. Because every enterprise maintains distinct architectural rules, coding standards, and internal conventions, an AI agent must parse existing legacy structures and apply changes that remain compatible with running services.
Multi-Tool Infrastructure and Task Resolution Rates
To mirror corporate environments, Real-SWE evaluates model-and-harness combinations instead of testing raw foundational models in isolation. The testing infrastructure integrates actual operational tools, where each task exposes only the specific services its workflow demands. The environment spans developer and runtime components including Docker, Kubernetes, AWS emulator, GitHub, Linear, MCP, PostgreSQL, MySQL, MongoDB, Redis, Gel, Go, Python, Node.js, Vitest, Slack, Intercom, and Google Drive.
One sample evaluation task requires resolving untaxed invoices in an environment utilizing a NestJS service, TypeScript, TaxJar sandbox, TaxJar production, and an InfluxDB ledger. Under this assignment, the agent must reconcile different company-specific tax calculation rules, query destination addresses against external tax authority providers via sandbox or production URLs, respect exemption flags, and log gross billing details into the ledger so returns reconcile without halting invoice delivery on failed address lookups.
Across these enterprise scenarios, resolution rates—measured as pass@1 averaged over eight independent runs per task—reveal a clear ceiling for existing frontier systems. Fable 5.1 running on Claude Code recorded the highest performance with a resolution rate of 38.8%, followed by GPT-6 Astra on Codex CLI at 33.8%, and Gemini 3.8 Flash on Gemini CLI at 31.2%. GLM 5.3 on Claude Code completed 28.8% of tasks, while Grok 4.6 on Grok Build and Muse Spark 1.3 on Muse Code tied at 23.8%. Kimi K3 on Kimi Code achieved 18.8%, and GPT-5.6 Sol on Codex CLI resolved 16.2% of its assignments.
Frontier agents capable of scoring at the top of synthetic benchmarks solve roughly one in three tasks when placed inside a real company's private repository. For business leaders, this performance drop signals an urgent need to recalculate return on investment and revise AI-assisted development strategies away from marketing claims toward hard operational metrics.