Serving large language models on on-premise infrastructure forces engineering teams to negotiate a painful trade-off: ballooning hardware budgets versus unacceptable latency. Speculative decoding promises relief by guessing speculative draft tokens before a larger target model verifies them in batched passes. Yet standard multi-token prediction (MTP) architectures stubbornly cling to fixed draft lengths—a rigid strategy that burns GPU compute on verification rollbacks the moment token entropy spikes.
To eliminate this inefficiency, developer stew675 submitted Pull Request #27210 to the ggml-org/llama.cpp repository, introducing dynamic draft depth via `--spec-type draft-mtp-adaptive`. The underlying mechanism relies on a finite-state machine pairing an ascension counter with a weighted drop-pressure accumulator.
"High depths fall quickly (a total miss adds N), low depths hold, and at the floor no pressure accumulates at all."
As developer stew675 documented in the pull request, draft depth climbs by one step after consecutive verification cycles accept all speculative tokens. Conversely, every missed prediction adds `(N - acceptance)` to the drop-pressure score, downshifting draft depth the instant accumulated penalty breaches threshold limits. At the base floor—configured by default to 3 through `--spec-draft-n-min-adpative`—pressure accumulation halts entirely, while higher draft regimes dynamically lower thresholds to accelerate forward depth expansion when predictions hit the mark.
Benchmark Performance Across Code and Prose Workloads
Empirical benchmarks conducted on Qwen3.8-27B Q8_0 across two Radeon AI PRO R9700 GPUs highlight tangible throughput gains on repetitive and highly structured tasks. The evaluation environment ran an 8192 context window, 8 execution threads, a temperature of 0.6, top-k of 20, top-p of 0.95, and min-p of 0.001.
In C++ code completion benchmarks generating roughly 750 to 800 tokens, a static draft depth of 2 delivered 67.2 tokens per second, while static depth 3 reached 78.8 tokens per second. Activating adaptive drafting with `--spec-draft-n-max 12` and a minimum floor of 2 pushed mean throughput to 86.4 tokens per second—maintaining operating depths between 10 and 11 despite an overall draft acceptance rate of 58.4 percent. Adjusting the baseline floor to 3 yielded 85.3 tokens per second across draft depths of 9 to 12.
Less predictable prose tasks naturally exhibited tighter performance boundaries, with static depth 3 stabilizing near 56.4 tokens per second. For infrastructure leads managing local deployments, PR #27210 demonstrates that algorithmic dynamism can unlock enterprise throughput gains and reduce total cost of ownership without purchasing additional accelerators.