The operational cost of serving language models remains chained to memory bandwidth, parameter representation, and unforgiving hardware constraints. While training ultra-low-bit architectures from scratch consumes massive compute budgets, post-training conversion promises to compress pre-existing checkpoints directly. In a technical pre-print, OneBit AI researchers Anirudh Malik, Poojith Devan, and M Sparsh Mehra evaluated an end-to-end post-training ternarization pipeline applied directly to the instruction-tuned Qwen3-4B.
Quantization Pipeline and Memory Footprint
The experimental workflow combines KOTMS rotation, E2M-ATQ ternarization, and GPTQ error compensation to generate a weight-only W1.58A16 representation while keeping model activations at full 16-bit precision. The conversion targeted linear weights, yielding an effective parameter budget of 1.641 bits per weight.
This aggressive parameter reduction translated directly into a dramatically smaller memory footprint during physical packing runs.
"compression is demonstrated; deployment acceleration remains an open engineering problem."
According to the OneBit AI benchmark data, lossless packing slashed the checkpoint from 8.29 GiB pre-packed down to a 3.96 GiB artifact. The mathematical lattice of ternary weights packs into half the storage volume without sacrificing the underlying precision of non-linear layers, opening a realistic path to running 4B-class models on budget consumer memory and low-tier edge hardware.
Accuracy Trade-Offs and Runtime Bottlenecks
Predictably, radical compression exacts a toll on model capability. Across ten evaluation benchmarks, average accuracy dropped from 64.5% in FP16 to 54.7% in W1.58A16—a 9.8 percentage-point tax. Language modeling perplexity followed suit: WikiText-2 perplexity climbed from 13.639 in FP16 to 18.748 in W1.58A16, with PTB and C4 exhibiting similar degradation curves.
The qualitative erosion proved starkly asymmetrical. Basic commonsense reasoning and context retention held their ground reasonably well, whereas dense, knowledge-intensive retrieval degraded sharply. More critically, shrinking the physical footprint delivered zero real-world execution speedup.
For engineering leadership, post-training ternarization proves that a 4B model can squeeze into a 3.96 GiB RAM envelope without an expensive retraining cycle. Yet until custom low-bit execution kernels mature, this approach trades substantial factual depth for RAM savings alone—a viable compromise for edge agents with constrained working memory, but a bottleneck for workloads demanding raw inferencing throughput.