Transitioning to low-precision inference has become an unavoidable pivot for organizations attempting to scale large language models without incinerating their compute budget. While the NVFP4 format promises a 4-bit floating-point structure with throughput that mocks FP8 or INT4, the industry's go-to recovery path—Quantization-Aware Distillation (QAD)—is hitting a structural wall. Engineers at PayPal have diagnosed a 'representation-level' failure that explains a frustrating paradox: why models that seem to mimic their high-precision teachers in output probability still fall apart when asked to solve complex reasoning tasks. The research by Fangbo Tu and Junhua Zhao reveals that standard KL-divergence loss functions act as a sophisticated mask, hiding the internal decay of a model’s semantic hierarchy under a veneer of correct-looking tokens.
The Failure of Output Matching in RL-Post-Trained Models
Traditional distillation is obsessed with the surface. It forces the student model to mimic the final output distribution of a frozen BF16 teacher, assuming that if the logits match, the logic must be sound. However, the PayPal team found that this 'output alignment' is a dangerous simplification. Using Centered Kernel Alignment (CKA) to probe layerwise similarity, the researchers discovered that standard QAD often deepens the representational drift it is supposed to fix. This phenomenon is particularly lethal in models shaped by reinforcement learning (RL). In these scenarios, the student model learns to cheat: it produces the 'right' tokens to satisfy the loss function, but it does so by mangling its internal activation geometry. The result is a model that passes superficial benchmarks but lacks the deep semantic stability required for enterprise-grade tasks.
Output matching alone can mask internal degradation, because many intermediate activation geometries can yield similar teacher-aligned logits.
This disconnect is why a 4-bit model might look competent in a demo but fail miserably in production-level coding or multi-step logical deduction. When the internal logic path is severed, the model isn't 'thinking' anymore; it is merely reciting. The study, which evaluated models like Qwen3-4B-Thinking-2507 and Nemotron 3 Nano, demonstrates that internal representational geometry is the only real foundation for robust low-bit generalization. Ignoring it during quantization leaves the model hollowed out, prone to hallucinations that no amount of output-level fine-tuning can fix.
CKA-QAD: Preserving Internal Geometry via Gram Matrices
To bridge this gap, the PayPal researchers introduced CKA-QAD, a methodology that treats the model's internal 'shape' as a first-class citizen. Instead of just grading the final answer, this approach aligns the layerwise Gram matrices between the teacher and the student. CKA is the surgical tool of choice here because it remains invariant to orthogonal transformations and isotropic scaling. Put simply: if the 4-bit constraints cause the data to rotate or scale within the activation space, CKA recognizes that the underlying logic remains intact. Traditional pointwise feature losses, by contrast, would panic at these benign shifts, forcing damaging and unnecessary corrections that degrade the model's intelligence.
CKA’s invariance to orthogonal transformations and isotropic scaling makes it less sensitive than pointwise feature losses to benign rotations and global scale changes in activation space.
By optimizing for both the output distribution and the internal geometric hierarchy, CKA-QAD allows the student to inherit the teacher’s actual reasoning process, not just its vocabulary. In tests on Nemotron 3 Nano and Qwen architectures, this method significantly clawed back accuracy in coding and reasoning benchmarks. For the C-suite, the business case is clear: it enables a radical reduction in inference latency and CapEx without sacrificing the 'reasoning' capabilities that make LLMs worth the investment. The training overhead for this alignment is negligible, making it a mandatory upgrade for any production pipeline eyeing 4-bit deployment.
PayPal’s findings signal a necessary shift from measuring what a model says to auditing how it arrives at a conclusion. While KL-divergence remains a standard, it is clearly insufficient for RL-post-trained architectures that require more than just word-matching. The industry must now face a new technical floor: if you aren't preserving the internal geometry, you aren't really distilling the model—you're just training a very expensive parrot. The next frontier will be determining if this geometric preservation holds at even lower bit-widths or if the NVFP4 format represents the final stable ground for complex latent spaces.