Infrastructure budgets for large language models are hitting a ceiling, but a fresh update to llama.cpp offers an elegant way out for those tired of subsidizing cloud providers. The merger of PR #25707 has introduced full CUDA support for ultra-low Q2_0 quantization (group 64). After being battle-tested on CPU and Metal, the technology has arrived for "green" hardware: NVIDIA GPUs can now efficiently process Ternary-Bonsai models ranging from 1.7B to 27B parameters.
Instead of chasing scarce H100s, businesses can pack weights into a ternary format (effectively 2 bits), radically lowering VRAM requirements. Data from llama-bench confirms the efficiency: on an NVIDIA L40S, the Ternary-Bonsai 27B model delivers a snappy 72 tokens per second while consuming only 7 GB of VRAM. For context, the 1.7B version accelerates to 530 tokens per second. This isn't just about cost-cutting; it’s about the ability to run serious neural networks on hardware previously considered fit only for basic display output.
Technical specifications and performance
Ternary-Bonsai 27B model: 72 tokens/sec at 7 GB VRAM consumption. Ternary-Bonsai 1.7B model: a record-breaking 530 tokens/sec. Quantization format: Q2_0 with group 64 (requires new GGUF builds). Compatibility: full CUDA backend support for NVIDIA graphics cards.
The technical integrity of the experiment is validated by KL divergence measurements: the difference between the "native" F16 format and the quantized Q2_0 is negligible. You get the same output logic at a fraction of the cost.
Key takeaways for IT architects
However, there is a catch regarding formats: the official CUDA backend is picky and requires new GGUF builds with the _g64 prefix. Older files from early forks using group 128 will not work here—you will need to update your local repositories.
While the market is drowning in marketing slogans about the need for total server upgrades, real optimization is happening quietly in GitHub repositories. This patch proves that for specific corporate tasks—from internal assistants to Edge AI—it is more profitable to squeeze the maximum out of existing racks rather than renting clusters. In our view, this is a prime example of how smart engineering overcomes hardware shortages.