Traditional LLM benchmarks suffer from a binary obsession: they tell you if a model failed, but they are useless at explaining why. According to Atul Anand and Sourav Chattaraj in their research 'Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools', current evaluations offer a black-box view of failure. To crack this open, the researchers introduced Canary Tools—diagnostic probes strategically planted within an agent's Model Context Protocol (MCP) toolset. These aren't just random distractors; they are engineered traps designed to expose specific cognitive distortions, turning a 'wrong choice' into a precise architectural diagnosis.
A Taxonomy of Tool-Selection Traps
The framework moves beyond simple noise by establishing a six-type taxonomy of failure modes. This includes semantic decoys that mimic the target tool's naming, parameter traps that test logic via argument constraints, and capability mirages—tools that promise a solution but lack the actual functionality. Other probes target prerequisite blindness, temporal decoys (outdated data), and granularity traps. As Anand and Chattaraj explain, each canary has 'known failure semantics,' allowing developers to map a model's mistake to a specific reasoning deficit rather than guessing at its internal hallucinations.
A canary is engineered so that the same wrong pick identifies which of six reasoning weaknesses occurred, turning the outcome into a typed diagnosis. By analyzing 8,640 task runs across 120 tasks, the study found that susceptibility to these traps drops sharply as models become more capable.
The findings are a reality check for the 'bigger is always better' narrative. While the susceptibility rate varied by 36x across eight tested models, with Llama 3.1 8B predictably struggling the most, the data reveals a non-linear relationship between cost and safety. In several instances, mid-tier hosted models were more prone to traps than their cheaper counterparts from the same provider. It appears that high training costs and 'frontier' branding do not automatically grant immunity to basic logic traps.
Performance Divergence in Model Context Protocol
The evaluation highlights a clear stratification of cognitive ability. Capability mirages—the most sophisticated of the traps—remained effective even against frontier models, while the other five canary types were mostly ignored by top-tier systems but consistently triggered by small 8B open-weight models. To ensure they weren't just measuring keyword matching, the researchers softened the 'giveaway' phrases in the canaries. The results held steady: frontier models' susceptibility didn't budge, confirming these probes measure genuine reasoning rather than superficial pattern recognition.
Capability tier alone does not predict safety: the most susceptible of the six hosted models is mid-tier, and within a provider the cheaper model can be the safer one. The research further found that canary susceptibility is a predictor of task failure, with a Spearman correlation of ρ=−0.34.
This research signals a necessary pivot for developers building autonomous systems. Measuring raw accuracy is no longer sufficient when deploying agents into complex API environments. The persistent vulnerability to capability mirages suggests that even our best reasoning models still struggle with the 'hallucination of competence.' For engineers utilizing the Model Context Protocol, the lesson is clear: expanding an agent's toolset without addressing its inability to parse prerequisites or granularity is a recipe for silent, systemic failure. Accuracy is a vanity metric; diagnostic depth is the new baseline.