Investigating the hyperfitting phenomenon reveals how compact pre-trained base models adapt when micro-datasets are trained down to near-zero training loss. In a study published on September 11, 2026, independent researcher tamewild evaluated small language models fine-tuned purely on 100 to 500 pure logic deduction traces consisting of 5x5 zebra puzzles, with zero mathematical data present in the fine-tuning set. The synthetic deduction traces were generated by Qwen3-235B-A22B-Instruct-2507 in September 2025 using rejection sampling.
Fine-tuning Qwen 3 4B Base on 100 zebra puzzles requires approximately 6.5 minutes on a single graphics processing unit. This brief optimization enables the 4B model to reach 85.26% on the full 5,000-problem MATH benchmark, marking a +31.16% delta over the base checkpoint and scoring 21.67% on AIME 2025.
Qwen 3 4B Base fine-tuned on 100 zebra puzzles reaches 85.26% on the full 5,000-problem MATH benchmark, delivering a +31.16% gain over base.
This behavior demonstrates that training on structured synthetic deduction steps transfers directly across domains, triggering generalized quantitative problem-solving capabilities without needing billions of domain-specific tokens.
Benchmark Performance Across Model Scales
The logic-driven transfer effect extends consistently across diverse model families and parameter sizes. Granite 4.1 3B Base fine-tuned on 500 zebra puzzles scores 77.73% on MATH-500 and 19.44% on AIME 2025, outperforming the official IBM Granite 4.1 3B Instruct baseline by 11.13 and 12.77 percentage points on those respective benchmarks.
Scaling the architecture up to Qwen 3.5 9B Base yields even wider gains when trained on 500 zebra puzzles. The 9B model advances from 17.00% to 80.50% on 5x5 zebra puzzles, with a Pass@3 metric of 97.05%. On the uncontaminated ArXivMath 05/26 evaluation, its accuracy climbs from 5.42% to 22.92% with a Pass@3 of 34.88%, while reaching 96.60% on MATH-500 and 60.67% on AIME 2025.
Adaptive Loss Scaling Through PCSS
To manage this micro-dataset optimization without instability, tamewild applied PCSS, or the Per-Example Calibrated Sigmoid Scaler. PCSS operates as an adaptive, data-aware loss scaler derived directly from the KTO framework.
The authors demonstrated that running KTO using only desirable/positive examples yielded results highly competitive with standard SFT.
As recorded in Table 4 of the original KTO paper, running the alignment algorithm exclusively on desirable positive examples matches conventional supervised fine-tuning. When configured with a beta parameter of 0.1, KTO increased the overall average benchmark score from 29.7% to 31.1%, driven by a substantial relative increase on the GSM8K mathematical reasoning benchmark from 1.0% to 12.5%.
Eliciting complex mathematical reasoning does not strictly depend on massive mathematical corpuses, as pure logic deduction datasets under adaptive scaling like PCSS can trigger latent algorithmic capabilities in compact architectures. For engineering teams, this signals a major shift in how custom AI agents can be adapted locally on minimal hardware without incurring staggering capital expenditures.