The era of primitive model routing is officially dead. For years, IT architects have built agentic systems based on a naive classification: send simple queries to cheap small models, route complex ones to flagships, and pocket the difference. However, a recent study by IBM Research (Yara Rizk, Eyal Shnarch, Jason Tsay, and Merve Unuvar) proves this routing logic is fundamentally flawed. What appears to be a model selection task is actually a system optimization problem that standard heuristics cannot solve.
The caching paradox and non-linear economics
The belief that a low price per million tokens guarantees a lower total cost of ownership (TCO) is crashing against the harsh reality of benchmarks. While analyzing 417 tasks in the AppWorld Test Challenge using a CodeAct agent, researchers discovered an anomaly: GPT-4.1 cost $155 ($0.37 per task), while Claude 3.5 Sonnet cost only $79 ($0.19 per task). On paper, this makes little sense: GPT-4.1 has lower input and output token costs, and Sonnet required three times as many reasoning steps for the same tasks.
A router that only looks at a vendor’s price list is optimizing the system based on arbitrary figures.
The solution lies not in neural network architecture, but in serving infrastructure. Agentic workloads reuse massive chunks of context at every step, making the cache hit rate the primary cost driver. Because Claude offered more aggressive discounts for cache reads, it proved more economical even with longer execution trajectories. Real efficiency now depends on the "model-workload-infrastructure" stack rather than marketing tables from OpenAI or Anthropic.
Invisible complexity at the gates
CTO attempts to categorize tasks by "complexity" are a dead end because complexity is invisible at the point of entry. As IBM Research notes, a prompt like "summarize this contract" might seem elementary but can trigger recursive searches, compliance checks, and calls to a dozen different tools. Conversely, a dense technical prompt might be resolved instantly by a specialized small model. Compounding this is a technical barrier: the latency and context degradation involved in passing "state" between different model architectures turn cascading routing into a financial black hole.
Optimizing for token price tags is yesterday’s strategy; it ignores the mechanics of modern agents. When the orchestration layer and caching logic dictate the final invoice, the model itself becomes a secondary variable. Today, true margins are created by edge-case systems engineering, not by squeezing discounts out of API providers.