The Mechanics of Spurious Advantage
Reinforcement learning with verifiable rewards (RLVR) has become the standard post-training paradigm for frontier reasoning architectures, with Group Relative Policy Optimization (GRPO) serving as the baseline advantage estimator across models like DeepSeek-R1. In a typical GRPO pipeline, a model samples a group of rollouts per prompt, scores them with a binary outcome verifier, and assigns advantage weights based on within-group reward statistics. As researchers Jiamian Wang and Zhiqiang Tao from Rochester Institute of Technology, alongside Samyadeep Basu, Koustava Goswami, and Tong Yu from Adobe Research, reveal in their latest study, this normalization mechanism treats flawed chains of thought that stumble into a correct answer identically to flawless deductive derivations. The estimator cannot differentiate genuine deduction from stochastic luck, introducing a systematic distortion termed spurious advantage that forces gradient updates to reinforce superficial guessing patterns.
Three Failure Points in RLVR Training
This spurious advantage destabilizes optimization across three distinct task architectures.
First, bounded-answer regimes like multiple-choice questions suffer immediately: random selection hits the target with a 1/N baseline probability, allowing entirely hallucinated steps to capture maximum positive advantage.
Second, nominally open-answer corpora with narrow intermediate sub-steps exhibit the same silent degradation whenever ungrounded shortcuts yield the correct final token.
Third, multi-turn search agents evaluated purely on final outcomes systematically learn noisy, redundant traversal strategies. Rollouts with larger action budgets explore wider search spaces and pile up invalid API calls or irrelevant intermediate queries while still landing on the gold answer. Because standard GRPO calculates advantage magnitudes directly from the ratio of correct to incorrect rollouts within each sampled batch, these lucky detours inject corrupted gradient signals straight into model parameters.
Eliminating Composition Dependence
To decouple deductive reasoning from random hits without ballooning compute budgets, the Adobe and RIT team engineered an alternative estimator dubbed SignBalance. Rather than computing per-rollout weights from intra-group distribution counts, SignBalance decouples advantage magnitude from the immediate ratio of correct to incorrect completions. The method preserves the binary verifier sign, enforces a global scaling constant, and restores zero-mean balance via a stop-gradient per-class rescaling step.
Across mathematical reasoning and search-agent benchmarks at multiple model scales, SignBalance matches GRPO on standard open-ended math while consistently outperforming it on bounded-choice tasks and complex search pipelines. For ML teams fine-tuning open reasoning architectures, the takeaway is unambiguous: outcome-only RLVR blindly over-indexes on answer matching unless gradient estimators explicitly neutralize lucky rollouts.