The industry’s obsession with feeding Large Language Models more data has hit a wall of diminishing returns. We’ve been operating under the assumption that hallucinations stem from a knowledge deficit—an 'empty shelf' in the model’s warehouse. However, new research from Google’s Nitay Calderon and Gal Yona suggests the crisis is actually one of logistics. In their study, “Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality,” the researchers argue that frontier models usually possess the facts; they just lack the mechanical reliability to retrieve them during inference. This shifting of the blame from encoding to recall suggests that scaling training data further is a brute-force solution to a surgical problem.
Identifying the Recall Bottleneck
To pin this down, the Google Research team built WikiProfile—a benchmark of over 2,000 Wikipedia-derived facts probed through ten different tasks. The goal was to determine whether a model truly 'knows' a fact (encoding) or can merely 'identify' it in a multiple-choice lineup without being able to produce it on demand (recall). The results are damning for the 'bigger is better' camp. Even when a fact is baked into the parametric weights, models frequently fail to access it during direct questioning. The study highlights a bizarre asymmetry: a model might correctly identify a person’s birthplace in a pre-training context but fail the same query when the prompt is reversed. It’s not that the model is ignorant; it’s that the inference path to that specific data point is broken.
Many factual errors in frontier LLMs are better understood as lost keys (recall failures), not empty shelves (encoding failures).
By classifying facts into five distinct profiles—ranging from total encoding failure to 'recall with thinking'—Calderon and Yona have provided a roadmap for why RAG and fine-tuning often feel like band-aids. If the knowledge is already there, adding more of it via fine-tuning won't fix the underlying structural inability to retrieve it. This realization forces a pivot in how we evaluate frontier models. We shouldn't just be measuring if the data was ingested; we need to measure how reachable that data is across diverse prompt geometries.
Thinking as a Tool for Knowledge Retrieval
Perhaps the most actionable insight from the report is the role of 'thinking' as a retrieval mechanism. The researchers found that some facts classified as unrecallable under standard conditions suddenly surfaced when the model used chain-of-thought processing. This implies that 'thinking' isn't just for logic—it’s a search query for the model's own weights. By eliciting intermediate steps, models can navigate their internal latent space more effectively, unlocking facts that direct questioning missed. For R&D leaders, this validates the push toward inference-time compute: if you want accuracy, you don't necessarily need a bigger model; you need to let the current one 'look' harder for what it already knows.
Encoding failures call for scaling model size or expanding data coverage, while recall failures might also point to post-training and inference-time methods.
The implications for corporate AI strategy are immediate. If the bottleneck is recall, the frantic race to build proprietary datasets for pre-training is secondary to perfecting post-training and inference-time optimization. We are moving toward a paradigm where 'accessibility metrics' will supersede raw accuracy scores. Future architectures will likely move away from monolithic scaling in favor of dedicated retrieval-optimized layers. The goal is no longer just to fill the warehouse, but to ensure the model never loses the key to the front door. Brute-force scaling is a legacy tactic; the future belongs to those who can make existing knowledge functional.