Academic natural-language-to-SQL benchmarks have spent years optimizing for clean, flat relational tables that look nothing like enterprise production databases. In practice, modern systems of record behave like graph-oriented, semi-structured data stores packed with polymorphic links, multi-level nesting, and typed arrays. When standard text-to-SQL models hit these real-world schemas, they collapse. Researchers from UT Austin and DevRev isolated this mismatch, exposing how standard baselines falter once relational schemas diverge from naive relational assumptions.
The Enterprise Benchmark Gap
Standard evaluation suites systematically mask production failure modes by testing against primitive-typed, flat schemas. Spider 2.0-Snow spans 7,860 tables across 152 databases, yet every single table relies strictly on primitive DDL definitions. Worse, barely 5% of tables contain descriptions, and nested-type traversals are virtually absent from its query distribution.
To establish an honest evaluation baseline, the researchers built DevRev NL2SQL, introducing 900 execution-verified queries mapped across nested types and link-graph structures.
"First, we introduce the DevRev NL2SQL benchmark: 900 execution-verified queries with nested-type and link-graph structure, accompanied by the Semantic Depth Score (SDS), a schema-agnostic rubric for analytical reasoning depth."
Alongside the corpus, the team introduced the Semantic Depth Score (SDS) to quantify analytical reasoning depth across enterprise workloads regardless of schema quirks.
Single-Generation Routing and Targeted Repair
Instead of burning compute on brute-force multi-candidate tournament consensus—an operational non-starter for low-latency enterprise pipelines—the architecture runs on a cost-aware single generation trajectory. The engine relies on dynamic metadata retrieval and iterative schema pruning before touching the model context window.
When execution errors occur, the pipeline skips vague retry prompts in favor of a deterministic error model. This translates runtime failures into targeted repair directives that fix lateral flattening bugs, ambiguous aliases, and broken type casting over nested attributes.
On DevRev NL2SQL, this single-generation agent hits 91.7% answer correctness, demolishing the next-best baseline by a 54.6 percentage point margin while slashing token spend.
Bridging the enterprise SQL divide does not require throwing more inference tokens at naive consensus voting. It demands schema-aware context filtering and deterministic error repair. How effectively this pipeline translates outside Snowflake-native syntax remains an open engineering challenge, but for enterprise data engineering teams, it proves that targeted metadata routing beats brute-force LLM compute on both accuracy and infrastructure cost.