Post-training has long operated under an expensive dogma: to instill rigorous reasoning into language models, developers had to supervise every single token along the logical path. The standard assumption was that without dense, step-by-step reinforcement across entire generated sequences, analytical coherence would collapse.
Rethinking Dense Feedback
Researchers Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, and Karim Bouyarmane from Amazon and Duke University have upended this baseline. Their findings show that robust reasoning can be unlocked using ultra-sparse supervision—intervening on as few as one or two tokens per trajectory during on-policy distillation (OPD), or roughly 0.05% of the total token volume.
"Reasoning can be effectively incentivized by an extremely small fraction of generated tokens—as few as one or two tokens per reasoning trajectory, corresponding to only 0.05% of all tokens."
Instead of paying the heavy computational tax of calculating gradients across full chains of thought, providing targeted feedback at critical leverage points matches—and often outperforms—traditional dense training. Sparse signals force the model to internalize the underlying logical structure rather than overfitting to token-level mimicry.
Cross-Architecture Validation
The research team confirmed this dynamic across nine teacher–student configurations spanning multiple model scales on mathematical and coding reasoning tasks, including evaluations on Llama architectures and Proximal Policy Optimization (PPO)-based reinforcement learning with verifiable reward (RLVR).
When testing mathematical reasoning on the AIME 2025 benchmark by distilling Qwen3-4B-Instruct-2507 into a Qwen3-8B student, the targeted `maxtok` sparse OPD variant produced a student model that surpassed both its teacher and standard dense OPD. Even `rand1tok`—a variant that randomly samples a single token per trajectory—significantly boosted reasoning over the base model, demonstrating that reasoning activation is far less dependent on continuous gradient guidance than previously assumed.
The Efficiency Gap in Post-Training
What this means for enterprise ML infrastructure: By slashing the volume of supervised tokens by 99.95%, enterprise engineering teams can drastically lower GPU cluster overhead during domain-specific distillation and fine-tuning. This unlocks a faster, cheaper path to deploying high-performing, compact local reasoning models without sacrificing analytical depth. The open operational question is how to generalize heuristic token selection—like `maxtok` and `mintok`—across messy, unstandardized enterprise workflows where verifiable reward signals are harder to construct.