A new pull request in the ggml-org/llama.cpp repository targets one of the most persistent bottlenecks in on-prem inference: the blunt instrument of whole-layer offloading. Submitted by contributor John-194 under PR #26622, the proposed `--n-cpu-ffn` flag allows infrastructure teams to selectively offload dense Feed-Forward Network (FFN) sublayers to system RAM, avoiding the dramatic throughput collapses historically triggered by standard `--ngl` layer-dumping.
The engineering logic mirrors the project's existing `--n-cpu-moe` routing. Rather than stranding entire transformer blocks in slower memory, operators can assign an exact count of parameter-heavy FFN blocks starting from index zero directly to host CPU cores. This surgical carve-out preserves precious VRAM for massive attention caches and sequence state without stalling compute pipelines.
According to benchmarks published by John-194 on GitHub, the architectural payoff is substantial. On a standard consumer test bench—an RTX 4070 Ti Super (16 GB VRAM), an Intel Core i5-13600KF, and 32 GB of DDR5 on Ubuntu 24.04 LTS—the hybrid pipeline sustained 15 to 25 tokens per second on a Q4_K_M quantized model while processing an active context window exceeding 90,000 tokens.
For enterprise infrastructure leads, sublayer-level splitting fundamentally alters hardware economics. Instead of requisitioning scarce, overpriced enterprise accelerators merely to accommodate swollen context buffers, teams can squeeze enterprise-grade long-context workloads out of existing CPU-heavy compute nodes. As John-194 noted in the implementation roadmap, future iterations targeting the largest FFN blocks first will only widen this efficiency dividend.