Optimizing local inference for Mixture of Experts (MoE) architectures remains a persistent bottleneck for engineering teams trying to escape high commercial API bills without blowing up their hardware budgets. Pull Request #26534 in ggml-org/llama.cpp directly targets this cost equation, implementing graph-level Multi-Token Prediction (MTP) for GLM-4.5-Air via the glm4moe graph_mtp execution path.
Technical Implementation and Architecture Support
Building on earlier work with GLM-4.7-Flash, this PR extends speculative decoding deeper into runtime execution graphs. The converter now handles `--no-mtp`, `--mtp`, and unified combined GGUF files, while the runtime loader supports combined, trunk-only, and standalone MTP-only layouts. For models distributed without an integrated NextN speculative block, developers can attach an external MTP-only draft head via Hugging Face. Crucially, validation runs confirmed zero regressions across adjacent architectures like Solar Open and GLM-4.6V.
Benchmark Performance Across Quantization Tiers
Benchmarked on a consumer-grade 4x RTX 3090 workstation, running fully offloaded `GLM-4.5-Air-trunk-only-Q4_K_M` alongside a `GLM-4.5-Air-MTP-Q4_K_M` draft generated an unassisted baseline of 74.55 tokens per second. Activating a single draft token (`--spec-draft-n-max 1`) lifted average throughput to 88.73 tokens per second—a 1.19x speedup—peaking at 94.41 t/s (1.27x) on structured reasoning like `stepwise_math` and 90.34 t/s (1.21x) on translation.
"Testing several existing GGUFs from Hugging Face (both combined-MTP and trunk-only) showed that the same external MTP draft also works with the tested finetunes, consistently providing a ~1.17-1.20x speedup with --spec-draft-n-max 1."
Pushing speculative depth further exposes hard architectural limits. Increasing to MTP 2 degraded mean throughput to 83.69 tokens per second (1.12x), while MTP 3 dropped performance below the baseline to 72.05 tokens per second (0.97x). For local MoE stacks, single-token speculation represents the optimal acceptance-rate sweet spot before draft validation overhead eats the margin.
Scaling Multi-Token Prediction to Full GLM-4.5
The optimization also scales to full-size GLM-4.5 deployments in memory-constrained environments, yielding a ~1.07x throughput gain even when offloading 30 MoE layers to CPU (`--n-cpu-moe 30`).
By embedding MTP support directly into the core runtime, open-source execution engines are systematically stripping proprietary cloud APIs of their speed advantage, enabling enterprises to deploy high-throughput local MoE models at a fraction of hosted infrastructure costs.