Frontier large language models can navigate intricate step-by-step reasoning on competitive benchmarks like MATH and GSM8K, yet they reliably stumble over grade-school arithmetic. A comprehensive survey led by Aoxin Ni at the University of Chinese Academy of Sciences highlights an uncomfortable truth: state-of-the-art architectures routinely hallucinate when handling multi-digit addition, magnitude comparisons, fractional values, and scientific notation.
The root breakdown is structural rather than a matter of scale. Autoregressive transformers process numbers as semantic text fragments rather than discrete mathematical entities. Byte-pair encoding (BPE) tokenization arbitrarily splits digits, while positional encodings fail to preserve spatial alignment for multi-digit calculations. Novel architectural patches that succeed when pretraining from scratch—such as Little-Endian formatting and Abacus Embeddings—fall flat when applied post hoc to frozen foundation models.
For enterprise engineering leads and fintech teams, treating an autoregressive model as a standalone calculator in quantitative pipelines is an operational failure waiting to happen. Brute-force weight fine-tuning cannot overcome fundamental tokenization bottlenecks; enterprise accuracy demands deterministic tool use, code interpreters, and hybrid computational scaffolding at inference time.